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In memory of Susan Loucks-Horsley 



During the last five years, Susan Loucks-Horsley did much of her work as part of the National 
Institute for Science Education. Susan was not only principal investigator of NISE’s professional 
development program, but a member of the leadership of NISE, joining with other team leaders to 
guide the Institute through its program of work. 

From the beginning of NISE, Susan Loucks-Horsley had a profound effect on our entire community. 
Committing to produce a book by the end of the first year of NSF funding seemed an ambitious — if 
not unattainable — goal. Susan and her coworkers did produce the book. It was a huge success and 
continues to have a profound effect on the thinking of people across the country as they design, 
deliver, and do research on professional development for teachers of science and mathematics. The 
book was NISE’s first “home run.” Susan’s having set and met such a goal inspired all of us in 
NISE to demand that level of accomplishment from ourselves. 

Susan was an outstanding scholar and researcher, of course, but she was also a person committed to 
making a difference through her scholarship. An expert professional developer, she was a master of 
communication skills and, more generally, interpersonal interactions. 

Although we will miss her professionalism, we will miss Susan as a friend even more. She was a 
wonderful, warm, and giving person. Susan made all of us better, both personally and professionally. 

— Colleagues from the National Institute for Science Education 
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Overview 



Mission 

The people and the work of the NISE share a 
strong and focused mission. We are 
committed to the improvement of mathematics 
and science education. Our scope includes 
practice from kindergarten through college, 
with attention to key transition points along 
the way. All of our work is directed toward the 
improvement of student learning; increased 
student learning is the ultimate test against 
which we judge our success. We recognize 
that mathematics and science education has 
disproportionately failed children from low- 
income and minority families and that this 
result must be changed for moral as well as 
economic reasons. We see education as in the 
midst of a technological revolution with great 
potential, but, as yet, little impact on practice. 
Our work seeks to capitalize on technological 
advances to create new forms of educational 
opportunity in mathematics and science for 
students of all backgrounds and varying 
interests. The NISE has ambitious goals. We 
tackle what we believe to be the toughest and 
most pressing problems. Nevertheless, we are 
committed to a high rate of productivity and 
an efficient and effective coupling of our work 
to the improvement of practice in real settings. 

When the Education and Human Resources 
Directorate of the National Science 
Foundation created the NISE in July 1995, it 
charged its first research center to “address the 
totality of the education enterprise, to assess 
its effectiveness and examine what new 
activities need to be established, what 
activities are no longer needed, and what new 
approaches will enhance science education.” 
Obviously, the NISE shares this broad and 
ambitious mission with a great number of 
other efforts supported by NSF and other 
federal agencies, as well as by foundations. 



professional organizations, and universities across 
the country. 

While the NISE was funded by NSF through a 
cooperative agreement, NISE was charged with 
identifying the substantive directions for its work. 
One of the major contributions of the NISE 
proved to be its ability to identify the most 
important and timely areas for science, 
mathematics, engineering, and technology 
(SMET) education R&D. The NISE architects 
recognized that in the EHR portfolio essentially 
no investment was being made to analyze and pull 
together large bodies of work in ways that would 
reveal emerging consensus. NISE not only 
completed much needed analysis of ripe bodies of 
research, but interpreted that work in ways that 
lead to productive development efforts serving the 
needs of practitioners (e.g., teachers, education 
administrators, professional organizations, federal 
agencies). Thus, from the very beginning, the 
NISE focused much of its resources on 
conducting analytic syntheses and meta-analyses 
of bodies of education R&D judged to be 
important and pressing and with sufficient work to 
make analysis and interpretation useful. 

Initially, professional development of K-12 
teachers was identified as an important and ripe 
target. Susan Loucks-Horsley’s team and the 
resulting book. Designing Professional 
Development for Teachers of Science and 
Mathematics, as well as their highly successful 
dissemination efforts, proved that the choice was 
good. NISE’s work led to a major 
reconceptualization of professional development 
and identified a new research agenda that has been 
taking K-12 professional development an 
important step forward. 

In 1995, EHR’s largest investment was in 
systemic reform, with programs for statewide, 
urban, and rural initiatives. The initial theory for 
systemic reform was more a hypothesis than an 
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established theory (Smith & O’Day, 1991). 

The NISE architects recognized the need to 
document and analyze the practices and 
effects in this mushrooming and rapidly 
evolving line of reform. Again, the decision 
proved to be wise. Bill Clune’s team 
extrapolated, from the wealth of data about 
practices and from the Smith and O’ Day 
hypotheses, an empirically based theory that 
has the highly desirable property that it can be 
empirically tested. At the same time, NISE 
discovered that systemic reform presents new 
challenges for evaluation, challenges not 
adequately met by current evaluation 
procedures. After five years, Norman Webb’s 
team has produced a promising book that 
documents the challenges of systemic 
evaluation and proposes new methodologies. 

NISE recognized the need for research-based 
tools to support the reform of first-year SMET 
courses in undergraduate education. These 
courses are the entry points for students who 
will pursue majors in SMET. They are the 
entry points for prospective K-12 SMET 
teachers. They are also often the last SMET 
experiences for college students, too many of 
whom become turned off to science and 
mathematics through bad experiences in their 
college coursework, and too many of whom 
are minorities and women. Again, NISE made 
a good decision. After five years. Art Ellis’s 
team has created a family of faculty 
professional development Web sites to support 
much needed undergraduate SMET reform. 
The development of each Web site began with 
a research synthesis, followed by a year-long 
institute, staffed by experts from across the 
country who identified promising practices 
and made them available to their colleagues 
through Web-based tools. Early response to 
the family of Web sites has been enthusiastic. 

Not all of NISE’s work has entailed big- 
picture, analytic synthesis of an existing body 



of work. We have also pursued narrower targets 
of opportunity for first-line empirical research. 
Two examples illustrate the point: The World 
Wide Web has revolutionized communication. 

The effectiveness of NISE has in large measure 
profited from Web-based communication, as 
illustrated by the Web sites to support reform in 
undergraduate SMET education and by the 
NISE’s own highly developed and carefully 
maintained Web site {http://www.nise.org). What 
NISE recognized was that (a) the Web might be a 
powerful tool for addressing science literacy and 
(b) creating a Web site in support of science 
literacy would also offer an excellent base for 
research on how Web-based learning occurs and 
how Web sites might be redesigned to better 
support learning. The result is the highly 
acclaimed The Why Files, created by Susan 
Trebach and Terry Devitt, which offers the 
science behind the news. Using the Web site as a 
launch pad, Sharon Dunwoody and William 
Eveland carved out a path-breaking program of 
research on Web-based learning. 

Because NISE’s mission requires an ongoing 
assessment of the most important SMET 
education R&D targets, it was only natural for 
NISE to assess continually the productivity of its 
own efforts. Early work on interdisciplinary 
problem solving, conducted by Sharon Derry, 
using NISE interdisciplinary teams as the focus of 
her study, was completed in Year 3. Early 
investments in formative evaluations by Susan 
Millar proved so useful in the beginning years that 
this effort could be scaled back. 

Freed up resources were redirected toward 
initiation of a new line of work on a Web-based 
approach to undergraduate teacher education in 
mathematics and science. Sharon Derry leads the 
effort. Many have recognized the inadequacies of 
current undergraduate teacher education programs 
to produce the kinds of teachers needed in K-12 
schools; how to overcome the weaknesses in 
undergraduate teacher education has been less 
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evident. Once again, NISE is utilizing the 
power of the Web to provide a whole new 
approach to undergraduate teacher education, 
one that might ultimately solve the highly 
problematic disconnect between 
undergraduate teacher education and the 
continuing professional development that 
teachers receive once leaving college and 
taking a teaching position. 

Figure 1 provides a schematic of NISE’s 
research program in the year 2000. While the 
mission of the Institute is broad, the Institute’s 
program is focused and coherent. Focus is 
necessary because resources are limited, and 
fundamental progress requires depth of effort. 
Coherence is important so that each separate 
line of work holds the potential for not only • 
achieving its goals, but informing the work of 
other lines of work. 

The three lines of work (see Figure 1) are tied 
together through a number of linkages to form 
a coherent focus for the Institute. Teacher 
education is arguably the single most 
important and most complicated piece of 
successful systemic reform. At the same time, 
college-level SMET education sets standards 
that drive the K-12 SMET education system. 
These higher education standards are key to 
any successful approach to K-12 systemic 
change. College-level SMET education also 
plays an essential role in defining the nature 
and quality of preservice teacher education. 
While preservice teacher education and 
professional development are not tightly 
coupled, either in concept or in practice, both 
would surely profit from such a linkage, which 
the Institute is seeking to make. A deep 
commitment to making the work accessible to 
the Institute’s target audiences permeates all 
activity. 

Three themes cross cut all of the work of the 
Institute. One is a deep concern for 



understanding how equity in SMET education can 
be achieved. Another is the role of technology in 
improving the quality of SMET education and 
providing strategies for implementing successful 
innovations on a national scale. The third cross- 
cutting theme is a commitment to dissemination. 
All lines of the Institute’s work — professional 
development and preservice teacher education, 
college-level SMET education, systemic reform — 
share a focus on problems of equity, profit from 
innovative uses of technology, and are committed 
to high-quality dissemination. 

Accomplishments 

NISE research on professional development in 
science and mathematics is a good example of the 
strength and influence of our work. Susan Loucks- 
Horsley’s 1994 book. Designing Professional 
Development for Teachers of Science and 
Mathematics, written with Peter Hewson, Nancy 
Love, and Kathy Stiles, is a top seller for Corwin 
Press. It has been put out twice in short form by 
the Eisenhower National Clearinghouse: once 
with a focus on science and once with a focus on 
mathematics. Tens of thousands of copies of each 
short form have been distributed to members of 
the field. The NISE Brief that presents the 
framework for professional development has been 
widely copied for use with practitioners in 
workshops and courses around the country. The 
work guides the current national evaluation of the 
Eisenhower Program. Clearly, the field needed 
this synthesis and reconceptualization of 
professional development in mathematics and 
science and gave it an enthusiastic reception. 

Soon there will be an NISE book that does for 
evaluation of systemic reform what Susan 
Loucks-Horsley’s book has done for professional 
development. The book by Norman Webb and 
others not only synthesizes the field, it pushes the 
field forward by reconceptualizing alignment, by 
making clear that the methodologies of program 
evaluation are not 
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Figure 1. Institute Research Program 
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adequate for coping with the complexities of 
systemic reform, and by offering alternatives. 
Also within the year, NISE’s book on a theory 
of systemic reform by William Clune and 
others will be completed. The real power of 
Clune’s theory lies with his ingenious 
operationalization of the variables. Empirical 
tests of validity can be conducted, and he has 
conducted the first. For systemic reform to 
remain useful, clarity and empirical tests are 
essential. 

NISE’s College Level One team, under Arthur 
Ellis’s leadership, is among the few groups in 
the country conducting serious research on 
how to improve the quality of learning, 
teaching, and assessment in undergraduate 
SMET education. Our early work on small- 
group cooperative learning began with a meta- 
analysis to see whether a research base existed 
for this often-promoted pedagogical strategy. 
We identified a large positive effect that has 
caught the attention of college teachers of 
science and higher education administrators 
across the country. Since then, we have 
extended the work to address classroom 
assessments, resulting in an exciting new Web 
site where SMET faculty can access powerful 
alternative assessment strategies for use in 
their instruction {www.wcer.wisc.edu/nise/ 
CLl). Current work is analyzing the impact of 
technology on student learning in 
undergraduate SMET education. Our research 
on undergraduate education and the resulting 
Web-based tools have been key to NISE’s 
successful involvement of scientists. 

As the above examples demonstrate, NISE is 
committed to research that steps back from the 
field and takes a big-picture view, pulling 
together what is known in new ways, 
sometimes leading to reconceptualizations, 
and always pointing to promising directions 
for practice and for future research. 



The NISE is equally committed to making its 
work influential. An unusually large percentage of 
resources is invested in connecting NISE’s work 
with its target audiences. One especially effective 
mechanism is the NISE forums. Under Senta 
Raizen and Ted Britton’s leadership, the NISE 
offered its first annual forum in its initial year of 
funding. The format was to select a rapidly 
developing area of NISE work, convene 300 or so 
leading practitioners and researchers in that area, 
and engage them in dialogue that results in a 
written statement capturing the current wisdom of 
the field. These invitational forums proved so 
successful that others, including NSF, 
increasingly looked to NISE to conduct forums on 
topics they identified or to advise them on how to 
run similar events of their own. The forum 
strategy is now widely available through a Corwin 
Press book. Designing Successful Meetings and 
Conferences in Education: Planning, 
Implementing, and Evaluating (2000). 

NISE has proven effective in producing highly 
visible and influential work. In five short years, 
NISE (a) built working partnerships with states, 
districts, scientific and education organizations, 
federal agencies, and foundations; (b) completed 
eight books; (c) published over 100 journal 
articles, research monographs, and briefs 
(including two articles in the most recent issue of 
the premier research journal in education, the 
Review of Educational Research, and a special 
issue of Teachers College Record), (d) produced 
eleven Web sites including the award-winning 
The Why Files (see http://whyfdes.news. 
wisc.edu/), which gives the science behind the 
news, now institutionalized in the UW-Madison 
Graduate School budget; and (e) conducted five 
innovative and highly successful forums (and two 
additional national meetings at the request of 
NSF). 



Strategies 

What accounts for NISE’s tremendous 
success, despite enormous challenges? During 
the course of five years of work, the following 
strategies have proven especially successful 
for the NISE. 

Getting the Best People 

Typically, education research centers place 
unnecessary boundaries around where they 
look for scholars to involve in the work. An 
early decision to commit NISE to 
interdisciplinary work has paid enormous 
dividends. NISE researchers come from the 
sciences, including mathematics; the social 
sciences; and education. Of the 95 individuals 
associated with the NISE leadership, 42 
percent came from education, 33 percent from 
SMET fields, and 15 percent from social 
sciences. 

NISE’s aggressive commitment to the strategy 
of interdisciplinary work can be seen in the 
following examples. When we created The 
Why Files, we went to the director of the 
University’s Office of News and Public 
Affairs, Susan Trebach, to lead that effort. 
When we saw the need for research on how 
individuals learn via the Web, we went to 
Sharon Dunwoody, professor of journalism. 
When we needed policy research on systemic 
reform, we went to William Clune, professor 
of law. When we sought to launch efforts to 
reform undergraduate SMET education, we 
went to Arthur Ellis, professor of chemistry. 
When we sought formative evaluation, we 
went to anthropologist Susan Millar. 

Neither has NISE limited itself geographically 
to the boundaries of the University of 
Wisconsin-Madison. When we launched our 
program of research on professional 
development, we went to Susan Loucks« 



Horsley, who was splitting her time between 
WestEd in Tucson, Arizona, and the National 
Research Council in Washington, DC. For 
leadership of our efforts in communication and 
dissemination, we went to Senta Raizen, forming 
a partnership with her National Center for 
Improving Science Education in Washington, DC. 

At the suggestion of EHR/NSF, we vigorously 
initiated a robust program of research fellows. 
Over the course of five years, 43 fellows joined 
NISE teams through varying arrangements and for 
varying periods of time. The fellows mechanism 
allowed the NISE to recruit the most capable 
scholars and practitioners in the areas on which 
NISE was focusing its efforts. Junior and senior 
scholars, practitioners, and members of the private 
sector have enhanced the quality of discourse and 
maximized the impact of NISE work. At the same 
time, NISE has provided opportunity and 
visibility to promising young scholars. 

Leadership 

Leadership is a key ingredient to the success of 
any organization, and NISE is no exception. 
Codirectors and associate directors have been 
used to provide complementary perspectives at the 
top leadership level. From day one, there has been 
parity between scientists and educators in NISE 
leadership. Initially, Denice Denton, an engineer, 
and Andy Porter, an education psychologist, were 
codirectors. In the fifth year. Porter, as director, 
was joined by associate directors Robert Mathieu, 
an astronomer, and Barrett Caldwell, an engineer. 
In the intervening years, Terrence Millar, a 
mathematician, served as codirector. The stability 
provided by Porter and project manager Paula 
White, and the new perspectives and new ideas 
contributed through changes from Denton to 
Millar to Mathieu and Caldwell, served to keep 
the NISE on course, with continuing renewal of 
ideas. 
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The most powerful leadership for the NISE 
comes from the Team Leaders Team. NISE 
early on was organized into research teams, 
one for each area of work. Each team is led by 
a senior scholar who makes NISE work a top 
priority. These leaders convene as the Team 
Leaders Team on a monthly basis. All 
evaluations of past progress and analyses to 
determine new directions are conducted by the 
Team Leaders Team. Rich substantive 
discussions of each team’s work at the team 
leaders’ table provide opportunities to learn 
from each other and to think about the work at 
the Institute level. 

Self-Evaluation 

NISE made a major investment in formative 
evaluation, led by Susan Millar, an 
anthropologist. The work of Millar and her 
team proved extremely valuable to the 
strength and well being of the Institute in its 
early years. In the first year, the formative 
evaluation identified some teams that were not 
functioning as well as expected, due to team 
leaders not having sufficient time assigned to 
the activity. At the beginning of year 2, time 
assignments were increased, and in one case a 
team leader was replaced. The result was 
dramatic improvements in productivity. Early 
problems in patterns of communication were 
detected and addressed. 

The NISE used two advisory boards as 
mechanisms for critical review, sometimes 
leading to redirection of efforts. A 17-person 
National Advisory Board consisted of a rich 
and balanced mix of scientists, education 
researchers, education practitioners, and 
representatives from business, industry, 
government, and foundations (see listing of 
membership in Appendix A). The National 
Advisory Board met each fall in years 2-5; 
each meeting focused on assessment of a 
specific aspect of NISE work. 



National Advisory Board meetings are expensive, 
with members coming from across the nation. 
NISE created an advisory team, essentially a local 
advisory board, comprised primarily of 
individuals in the Madison area, most of whom 
were professors at the University of Wisconsin- 
Madison. This group proved to be especially 
useful in reviewing the quality and quantity of 
each team’s productivity and in reviewing the 
Institute’s draft plans for new areas of work to 
pursue. The local advisory board, which met 
quarterly, required no travel costs, and all 
members volunteered their time. 

EHR/NSF made a third-year review part of the 
initial cooperative agreement. This, too, proved to 
be a valuable tool for criticism and reflection. In 
preparation for NSF’s third-year review, NISE 
used both its national and local advisory boards to 
review the productivity of each team and to help 
the NISE develop plans for where the work should 
go in the final two years. The investment in 
formative evaluation was decreased; the work on 
interdisciplinary problem solving was completed; 
and work on undergraduate teacher education was 
begun. The third-year review also led to 
combining the systemic policy and systemic 
evaluation teams into one integrated effort. For its 
part, the NSF completed both a site visit and a 
reverse site visit for the third-year review. 

Additional Funding 

EHR/NSF set the budget for NISE at $2 million 
per year for each of five years. The plan for level 
funding proved to be a mixed blessing. Any 
productive institute needs progressively more 
financial support as it becomes established and 
matures. Level funding, in that sense, inhibits 
effective progress. Recognizing that level funding 
would soon become a problem, NISE leadership 
put in place three strategies. 

First, NISE attempted to manage its NSF funding, 
spending less in the early years so that, through 
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carryover, more funds would be available in 
the later years. This was an error in judgment 
as NSF did not permit carryover from year 4 
to year 5 of the Institute. NISE’s second 
strategy was more successful, moving 
activities off its budget and onto the budgets 
of other organizations where ongoing activity 
could be sustained. One successful 
implementation of this strategy was The Why 
Files. In year 3, the Graduate School at the 
UW-Madison took over the effort; the 
Graduate School will continue The Why Files 
for the indefinite future. Another successful 
implementation of this strategy concerns the 
Web sites to support reform of undergraduate 
SMET education. In particular, the Web site 
supporting the improvement of classroom 
assessment has been spun off with continued 
NSF support to the University of New 
Mexico, under the leadership of NISE Fellow 
Michael Zeilik. 

Yet a third strategy NISE used to deal with 
level funding was to secure additional support 
from other sources. For example, in year 5, in 
addition to the $2 million from EHR/NSF, 
NISE received $2 million from a variety of 
other sources. Not only has this strategy been 
especially effective in solving the level- 
funding problem, but it positioned the NISE 
for continuing beyond its fifth year, even 
though EHR/NSF decided not to extend 
institute-level funding through the mechanism 
of a recompetition. 

As an institute, NISE has evolved a unique 
and productive approach to pursuing work. 

We collaborate in interdisciplinary teams with 
a rich mix of scientists, education researchers, 
and educational practitioners. We employ a 
vigorous fellows program to bring a diverse 
group of both junior and senior scholars, from 
across the nation and around the world, to join 
our interdisciplinary teams and to give our 
work a national base. We see the interplay 



among research, development, and practice as 
multidirectional and interactive; each is dependent 
upon insights and knowledge from the other two. 
Our commitment to improving student learning in 
real settings has brought us to a strategy of 
producing useful and effective tools to place in the 
hands of practitioners; the World Wide Web has 
been an especially effective medium in that 
regard. We make a large and intense investment in 
dissemination, using multiple and complementary 
strategies. Our innovative approaches to 
dissemination include Web-based tools, special 
relationships with collaborating professional 
organizations, and our unique annual forums, 
together with more traditional approaches of 
books, journal articles, research monographs, 
occasional papers, and workshop series. We 
produce timely, widely circulated briefs on 
pressing topics. 

Making the Institute work requires a powerful and 
robust infrastructure. Over the course of the last 
five years, our management structure has 
developed into an efficient and effective force for 
keeping work on time and of the highest quality — 
and for producing a multitude of products. We 
have evolved into a productive and unique 
community of scholars, with ongoing cross- 
fertilization among teams and with our target 
audiences. This community of scholarship serves 
as a hotbed for identifying new initiatives and 
ideas. The Institute seeks to continually reinvent 
itself by (a) spinning off successful developed 
products so that they can take on a life of their 
own through other support, (b) bringing to a close 
lines of work that have fulfilled their intentions, 
and (c) initiating new lines that hold greatest 
promise for accomplishing our mission. 

Challenges 

The NISE overcame a number of serious 
challenges during its initial five years. 

Being the first center-level activity of the EHR 
and funded through a cooperative agreement. 
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NISE had to negotiate with EHR the 
procedures for working together. Initially, all 
of the EHR divisions contributed to NISE 
funding. Understandably, each wanted to have 
a say in what work the NISE would undertake. 
Coming to agreement about what work would 
be done in a timely fashion proved to be 
challenging. Even after EHR funding was 
consolidated in the Research, Evaluation, and 
Communication Division (REC), frequent 
changes in leadership at REC made building a 
functional relationship difficult. 

Two strengths developed over time in the 
relationships of NISE to NSF. First, we had 
the same project officer throughout the five- 
year period, Larry Suter. A knowledgeable 
education researcher himself, and an effective, 
dedicated federal official, Suter sustained a 
positive, useful relationship with NISE. He 
understood the importance of NISE work and 
became an advocate for the Institute at the 
EHR leadership table. Second, NISE 
developed excellent working relationships 
with other divisions within EHR. For example, 
the College Level One Team, reforming 
undergraduate SMET education. 



worked closely and productively with the Division 
of Undergraduate Education. The Professional 
Development Team worked closely and 
productively with the Division of Elementary, 
Secondary, and Informal Education. Finally, 
because of the quality of NISE work, because of 
NISE’s increasing visibility in the field, and 
because of NISE’s good relationships with the 
other divisions within EHR, NISE established a 
positive relationship with then EHR Director 
Luther Williams. 

The final challenge to NISE was EHR’s decision 
not to recompete the Institute for an additional 
five years of funding. A center is a difficult and 
complicated organization to create and bring to 
productivity. Five years of support is simply too 
short a period for such a fragile but valuable 
capacity. An additional five years would 
definitely have been a wise decision on EHR’s 
part and would have resulted in maximum 
utilization of the initial five-year investment. Even 
without NSF institutional support, the NISE has 
decided to continue as an institute. At this point, it 
is too early to tell whether we will be successful, 
although funding for at least one additional year is 
in hand at this writing. 
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Research Programs 

Systemic Reform: Policy and Evaluation 



The objective of the Systemic Reform Team, 
codirected by William Clune and Norman 
Webb, is to make a significant contribution to 
the study of systemic reform, including how 
knowledge about it can be strengthened in the 
future. The present combined team was 
formed by consolidating the efforts of two 
previously independent research teams. Policy 
Analysis of Systemic Reform (PASR) and 
Strategies for Evaluating Systemic Reform 
(SESR), and has produced a new body of 
integrated research. Through the work of team 
members, outside experts, conferences, 
forums, and fellows, we have generated a 
knowledge base that has been presented in 
joint activities and research papers and will 
soon culminate in production of two books, 
one on the theory of systemic reform and the 
other on the challenges to evaluating systemic 
reform in mathematics and science. We have 
successfully built on our core NSF work to 
obtain outside funding for studying systemic 
reform and building district capacity in the 
Milwaukee Public Schools (funded by the 
Joyce Foundation, the Helen Bader 
Foundation, and the Spencer Foundation) and 
for studying the impact of the Statewide 
Systemic Initiatives (funded by the National 
Science Foundation). These projects 
complement the research we have done with 
core NSF support by creating a synergy 
between confronting and solving very 
practical challenges while also being reflective 
and theory-driven in analyzing systemic 
reform as a strategy for large-scale change. 



Policy Analysis of Systemic Reform 

I. Policy Team Mission 

The Policy Analysis of Systemic Reform 
component of the Systemic Reform Team seeks to 
generate useful knowledge about implementing 
and improving the effectiveness of systemic 
reform in mathematics and science education. The 
Policy Team focuses on meta-analyses and 
syntheses of existing knowledge about systemic 
reform in SMET education, including research, 
evaluation, and the wisdom of the practitioner. 

II. Accomplishments 

Convening of Scholars and Practitioners 

The early research conducted by the Policy Team 
culminated in a “summer seminar” held July 25- 
26, 1996, in Madison, attended by a broad group 
of reformers, policy analysts, evaluators, and 
experts in the content standards and their subject 
matters. William Clune (1999) wrote a research 
monograph synthesizing the problems identified 
as central in the seminar and suggesting how the 
future research agenda could be shaped to deliver 
the most useful knowledge about reform. Clune's 
monograph deals with two general issues: (1) how 
to think about systemic reform as an effort to 
produce substantial gains in student achievement; 
(2) how to design the research to produce 
knowledge that is useful in improving systemic 
reform in mathematics and science education. 
Michael Knapp (1997) wrote a research 
monograph on the implementation of systemic 
reform; Michael Kirst and Robin Bird (1997) on 
the politics of standards; and several authors 
(policy analysts, mathematicians and scientists) 
wrote papers on the national standards themselves 



11 




16 



(Clune, Haimo, Roitman, Romberg, Wright, & 
Wright, 1997). The set of papers on the 
national standards resulted in a special issue of 
Teachers College Record (1998, Fall). Senta 
Raizen (1997) prepared an NISE Occasional 
Paper on standards for science education that 
provides the current rationale and historical 
background for science standards, discusses 
the various meanings and interpretations 
associated with the standards, and addresses 
the current status of science curriculum 
frameworks including commonalities and 
variations among them. In addition, the NISE 
Occasional Paper prepared by Michael Kirst, 
Robin Bird, and Senta Raizen (1997) uses 
historical and recent concerns about 
mathematics and science content standards to 
demonstrate conflict and tensions that 
surround the process of setting standards. 
While subject matter specialists are an 
important component of any resolution, this 
Occasional Paper stresses the multiple 
perspectives that must be considered. 

Four “systemic seminars” were convened in 
Madison for the purpose of gathering more 
information relevant to the research agenda 
from written research and discussion with 
NISE and NSF personnel. The systemic 
seminars brought together small groups of 
reformers, scholars, and evaluators with in- 
depth knowledge of reform. The small groups, 
which included people from Statewide and 
Urban Systemic Initiatives (SSIs and USIs) as 
well as the NSF evaluation personnel, 
addressed such issues as the proper criteria for 
evaluation and what works and does not work 
in systemic reform. This phase of developing 
the research agenda yielded a new set of 
refined links between process and outcomes in 
systemic reform that guided further research. 

The theme of the 1 997 NISE Forum was 
“Research on Systemic Reform: What Have 
We Learned? What Do We Need to Know?” 



The goal of the Forum was to generate and share 
knowledge of systemic reform, including the role 
of curriculum, the role of teaching and learning, 
and the role of evaluation. To reach this goal, 
panelists from around the nation shared their 
expertise in these various areas with participants, 
who themselves brought varied expertise to 
structured conversations and networking sessions. 
The Forum resulted in a set of papers identifying 
important issues and a synthesis paper of the think 
pieces that the participants submitted around the 
central issues (Clune, Millar, Raizen, Webb, 
Bowcock, Britton, Gunter, & Mesquita, 1997). 

Development of a Theory of Systemic Reform 

The Policy Team conducted a telephone survey of 
the staff of all SSIs funded by NSF. An interview 
protocol on the history of each initiative and 
lessons learned for policy was developed, and 48 
telephone interviews were conducted (involving 
22 states, 21 principal investigators, 22 program 
directors, and 5 others). Documents from each 
state were collected (for example, proposals, 
annual reports, program evaluation reviews). 
Based on the interviews and documents, 
summaries of individual SSIs were prepared. 
Analysis of these summaries was included in a 
symposium conducted by the Policy Team at the 
1999 annual meeting of the American Educational 
Research Association. Comprehensive analysis 
will be included in the book on the theory of 
systemic reform. 

Members of the Policy Team met regularly to 
discuss the results of the survey and develop a 
theory of systemic reform that could be tested by 
the accumulating data. Initial formulations of the 
theory and examples of how it might be used to 
guide analysis of data on SSIs were presented at 
an NISE Brown Bag Lecture. The theory that 
emerged from these activities has provided the 
framework for a book on systemic reform. Theory 
and Practice of Systemic Reform of Mathematics 
and Science Education. The theory presented in 
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the book will contribute to the knowledge of 
reformers, policymakers, researchers, 
practitioners, and educational administrators 
about the design, implementation, and effects 
of systemic reform and represents the only 
attempt to date to concretely specify the full 
range of substantive and strategic factors that 
drive successful systemic reform. An NISE 
Research Monograph is currently available on 
systemic reform theory (Clune, 1999). 

Evaluation of Systemic Reform 

I. Evaluation Team Mission 

At its outset the Strategies for Evaluating 
Systemic Reform Team developed an overall 
sequence of research and development. It set a 
general progression for its work over the five 
years beginning with (1) identifying the major 
issues and questions about evaluation of 
systemic reform followed by (2) outlining 
specific strategies and models for evaluating 
systemic reform and then finally (3) putting 
into practice what we learned. Over the five 
years our work has followed this general 
direction and has become more focused with 
the joining of the evaluation and policy teams. 
What we did not anticipate in 1995 at the 
beginning of this effort was the nature or the 
magnitude of our current efforts to advance 
the study of large education systems. 

In parallel and in cooperation with the Policy 
Team, the overall mission of the Evaluation 
Team is to advance knowledge on evaluating 
large systems. The Evaluation Team has 
striven to expand the existing knowledge base 
on systemic evaluation. We have done so by 
accumulating, synthesizing, and producing 
knowledge about systemic evaluation along 
with developing models for doing evaluation 
of large systems. Our work has drawn from 
the practitioners who have engaged in 
evaluating systemic reform; scholars who 



have studied change in large education systems; 
the insights and work of fellows; an 
interdisciplinary team of scientists, 
mathematicians, and educators; literature on 
systems and measuring change in education; and 
our own research. Our work continues into the 
future by applying an embedded research model 
to build capacity in Milwaukee Public Schools 
(funded by a second two-year grant from the 
Joyce Foundation from September 2000 through 
August 2002); developing models for analyzing 
student achievement in large education systems 
(funded by a one-year grant from NSF from May 
2000 through April 2001); and analyzing the 
impact of systemic reform on states (funded by a 
three-year grant from NSF from January 1999 
through December 2001). 

II. Accomplishments 

Convening an Interdisciplinary Team of Scholars 
and Practitioners 

A working group of scholars and practitioners met 
over two years to launch the study of evaluation 
of systemic reform. This group included 
Christopher Anderson, professor of astronomy; 
Vicki Bier, professor of industrial 
engineering/applied mathematics; Steven 
Bauman, professor of mathematics; Thomas 
Carpenter, professor of mathematics education; 
Donald Chambers, former state supervisor of 
mathematics education; Susan Millar, 
anthropologist and evaluator; Senta Raizen, 
science educator and evaluator; Thomas 
Romberg, professor of mathematics education; 

Pat Rossman, elementary science teacher and 
1994 Wisconsin Teacher of the Year; John Witte, 
professor of political science; and John Wright, 
professor of chemistry. This group met bimonthly 
to help identify the major issues that should be 
addressed in evaluating systemic reform, the 
questions that should be addressed by an 
evaluation of systemic reform, and the profound 
changes in student learning and important 
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knowledge in SMET. Members of the working 
group actively participated in two conferences 
that included discussion of papers and small 
working groups to identify important design 
features of evaluations of systemic reform. 

The working group was very valuable in 
informing the early thinking about evaluation 
of systemic reform. Members of the group 
identified knowledge necessary for success in 
higher education, qualities of education not 
necessarily assessed by standard measures of 
achievement, equity issues in a male- 
dominated field such as engineering, the 
difficulty in identifying prerequisite SMET 
knowledge in a range of higher education 
courses, and the importance of using value 
added analysis of standardized norm- 
referenced tests. 

Networking and Conferences 

Networking among evaluators of NSF’s SSIs 
was encouraged by convening two conferences 
and holding other small focus meetings. 
Twenty-six people from across the nation 
attended the first conference held in Madison 
on January 4-5, 1996. Four papers were 
prepared for this conference to direct the 
discussion. Kathy Comfort, then of the 
American Association for the Advancement of 
Science, prepared a paper titled “Student 
Outcomes in Science.” Jim Ridgway, 
University of Lancaster, England, and Hugh 
Burkhardt, Shell Centre of Mathematical 
Education, England, prepared two joint papers, 
one entitled “Student Outcomes in Evaluating 
System Change” and the other entitled “System 
Alignment and System Change.” Andrew 
Jackman, Mt. Hood Community College, 
Oregon, wrote a paper on assessment and 
educational reform that raised many questions 
about education reform, particularly as it 
pertains to community colleges. 



One year later, on March 13-14, 1997, over 50 
people attended a conference in Madison on 
systemic evaluation. Attendees included SSI and 
USI evaluators, working group members, NSF 
staff, other evaluators, and other NISE staff. The 
conference successfully met its goals: strengthen 
the intellectual qualities of the research 
monographs and papers of the evaluation working 
group and its fellows, further the networking 
among those engaged in doing systemic 
evaluations, and disseminate NISE work to others. 
Besides hearing presentations from four 
evaluators of SSIs and John Witte, a UW- 
Madison political scientist, the participants 
provided feedback to NISE Fellows who were 
preparing papers (Bruckerhoff, Kahle, Lee, and 
Ridgway) and gave their thinking on designing 
systemic evaluations. The participants gave 
cogent advice on questions about outcomes, 
education systems, evaluation, issues related to 
design, and characteristics of evaluation designs. 
In addition to the two conferences, on two 
occasions SSI evaluators and other researchers 
were invited to a small focus meeting to discuss 
their work and insights. Daniel Heck and Norman 
Webb (2000) wrote a monograph synthesizing 
information learned from these conferences, 
meetings, and the evaluations of SSIs. NISE 
Fellow Ridgway contributed to advancing 
thinking about systemic reform by synthesizing 
information from a number of fields. His 
monograph on modeling of systems and 
macrosystemic change (Ridgway, 1998b) drew on 
the sciences, learning theory, epidemiology, and 
ecology to distinguish among different system 
models and indicate ways for building the 
evidence base for evaluating systemic reform. 

Alignment Criteria Studies 

In close cooperation with the Council of Chief 
State School Officers (CCSSO) and through a 
grant to CCSSO from the National Science 
Foundation, Webb completed an alignment study 
of state standards in mathematics and science and 
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the state large-scale assessments in each of 
four states. This analysis was based on 
Webb’s NISE monograph on alignment 
(1997a). An NISE brief on alignment was 
published in January, 1997. At the end of June 
1998, 15 reviewers attended a four-day 
institute in Madison and applied four 
alignment criteria identified in the NISE 
monograph. An important purpose for this 
institute and the analysis of the data was to 
refine the process forjudging the alignment 
between standards and assessments. The 
coding by reviewers was tabulated and 
summarized. The report on each state 
describes the degree of alignment between 
standards and the corresponding assessments 
on the criteria of categorical congruence, 
depth-of-knowledge consistency, range-of- 
knowledge correspondence, and balance of 
representation (Webb, 1999c, d, e, and f). 

The summary report presents the major 
findings and the recommendations for 
improving the coding and analysis process 
(Webb, 1999a). Based on the analyses 
performed, clear differences among the states 
were evident along with common issues faced 
by all. A high percentage of standards and 
assessments across the four states failed to 
achieve depth-of-knowledge consistency. In 
general, too many items were below the 
depth-of-knowledge level of the 
corresponding objectives for there to be 
alignment. One benefit found for doing an 
analysis using the specific criteria is that 
specific feedback was provided to states on 
what could be done to improve alignment. The 
criteria and the underlying structure of the 
analysis proved to be viable for detecting the 
degree of alignment and how alignment can be 
improved. 

Based on his work on alignment through NISE 
and CCSSO, Webb made presentations at a 



number of conferences (American Educational 
Research Association; CCSSO Large-Scale 
Assessment Conference; Rand Conference on 
Assessment; National Center for Research on 
Evaluation Standards and Students Testing 
(CRESST); Abo Akademi University, Vasa, 
Finland; and the National Council of Teachers of 
Mathematics). He also served as a consultant, 
advised, or conducted training on alignment for 
the Department of Education (Title I), Indiana 
Mathematics Initiative, Achieve, Inc., and the 
American Association for the Advancement of 
Science (AAAS). 

Book on Evaluation of Systemic Reform 

Two NISE Fellows, Jeanne Rose Century 
(Education Development Center, Newton, 
Massachusetts) and Norma Davila (University of 
Puerto Rico), and Evaluation Team staff Webb, 
Heck, and Osthoff have completed 13 of 19 
chapters of a book. Evaluation of Systemic Reform 
in Mathematics and Science. The work on this 
book paralleled and interacted with work done by 
the Policy Team on its book. Theory and Practice 
of Systemic Reform of Mathematics and Science. 
The evaluation book will serve as both a practical 
and a theoretical guide. Systemic reform is still a 
theory waiting for an accumulation of evidence 
that will make the theory credible. The book 
details an approach to evaluating systemwide 
education reform at a state, district, or school 
level. Central to this approach, and the centerpiece 
of the book, is studying and measuring an 
education system’s progress on the basis of nine 
attributes. These attributes cluster into three 
groups that are distinguished by how the attributes 
relate to systemic reform and to improving student 
outcomes within the system. A prospectus has 
been submitted to two publishers for their 
consideration. One publisher expressed interest, 
but has not issued a firm commitment. 
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Synthesis on Evaluation of Systemic Reform 

On February 1 and 2, 1999, the Fourth Annual 
NISE Forum was held. This forum was 
devoted entirely to evaluation of systemic 
reform. Over 260 people attended from a wide 
range of organizations including professional 
organizations, universities, school districts, 
states, government agencies, corporations, and 
foundations. The Forum was organized into 
five sessions including the overview of the 
conference given by Porter and Webb from 
NISE and John Hunt from NSF; a panel on 
understanding evaluation of systemic reform 
(chaired by Bernice Anderson of NSF and 
including Juanita Clay Chambers [Detroit 
Public Schools], Daniel Heck [University of 
Illinois], Zoe Barley [Western Michigan 
University], and Iris Weiss [Horizon 
Research]); a panel on models and approaches 
to evaluation of systemic reform (chaired by 
Larry Suter of NSF and including Patrick 
Shields [Stanford Research International], 
Jeanne Rose Century [Education Development 
Center], Norma Davila [University of Puerto 
Rico], and Mark St. John [Inverness 
Research]); and a panel on findings about 
systemic reform from evaluations and research 
(chaired by Julio Lopez-Ferrao of NSF and 
including Jane Butler Kahle [Miami 
University], Daryl Chubin [NSF], Robert 
Meyer [University of Wisconsin-Madison], 
and Clune [NISE]). After each panel, 
participants were divided into groups of 8 to 
10 for breakout sessions of focused discussion 
on the topic of the proceeding panel. Ernest 
House (University of Colorado) and Cora 
Marrett (University of Massachusetts- 
Amherst) gave their syntheses of the 
conference considering the presentations and 
small group discussions. Marshall Smith, U.S. 
Department of Education, gave summary 
remarks. Evaluation of Systemic Reform in 
Mathematics and Science: Synthesis and 
proceedings of the Fourth Annual NISE 



Forum, includes papers from all of the presenters 
(Webb, 1999b). 

Criteria for Evaluating Education Information 
Networks 

Information systems are arising as an essential 
component of systemic reform. Reform itself and 
the evaluation of improvement in student 
achievement are greatly hampered by insufficient 
data, fragmented data bases, and low knowledge 
of how to use data to make important decisions 
necessary to improve student achievement. One of 
the key findings from our studies is that states, 
districts, and schools do not have data in a form 
that can be used to monitor progress toward 
systemic reform. In 1998, the Evaluation Team 
began a discussion group focusing on InfoNets, 
information networks for education systems. The 
multidisciplinary group included Susan Daffinrud, 
master’s degree in both mathematics and 
computer science; Christopher Thom, director of 
WCER technical services; Barrett Caldwell, 
professor of industrial engineering; Jeffery 
Watson, graduate student in industrial 
engineering; Susan Zeyher, master’s degree in 
library science; and Jeffery Choppin, former high 
school teacher and graduate student in 
mathematics education. 

We inventoried existing products and activities 
that are being advanced for processing and 
meeting the multiple functions for an education 
information system. On June 28-29, 1999, 18 
people attended a conference to identify current 
electronic network systems and to help plan what 
role NISE should serve. Attendees included 
electronic system developers, a researcher in 
technologies, computer systems consultants, 
district technology staff, an educational 
psychologist, and NISE staff. Based on the 
outcome of this meeting and other discussion, we 
began to develop a framework that could be used 
to evaluate and analyze electronic information 
systems used in education. Thom (2000) wrote a 
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paper on knowledge management for 
education information systems. Watson and 
Zeyher (2000) wrote a paper on education 
system information needs and systemic 
reform. 

The Joyce Foundation made a two-year grant, 
August, 1999, through July, 2001, to study 
electronic information systems in schools in 
general and the trial of Quality School 
Portfolio in particular. We are conducting this 
study in six schools in Milwaukee in 
coordination with our study to develop 
analytic capacity in the district. 

Dissemination of Information on Evaluation of 
Systemic Reform 

At the American Evaluation Association 
annual meeting in San Diego, November 6, 
1997, a panel on the role of evaluation in 
education systemic reform was organized by 
Webb based on the NISE conference held in 
March 1997. Panelists included Lloyd Bond 
(University of North Carolina, Greensboro), 
Zoe Barley (Western Michigan University), 
and Patrick Shields (SRI International). At the 
American Educational Research Association 
annual meeting in San Diego, April 16, 1998, 
Webb and NISE Fellows presented a 
symposium on evaluating systemic initiatives. 
NISE Fellows who presented were Ridgway, 
Kahle, and Lee. Joan Herman, 

CRESST/UCLA, was the discussant. At the 
American Evaluation Association annual 
meeting, November 7, 1998, in Chicago, 
members of the systemic evaluation writing 
team gave a session on issues in evaluating 
student achievement in the context of systemic 
reform. Presenters were Webb, Davila, Heck, 
and Osthoff with Barley providing reactions. 

At the American Educational Research 
Association annual meeting, April 23, 1999, in 
Montreal, members of the writing team gave a 
session entitled Beyond Tradition: The Realm 



of Systemic Educational Reform. Members of the 
writing team who participated included 
LeMahieu, Heck, Century, Davila, and Webb. 
Barley served as the discussant. 

Accomplishment 8. Directorate of Education and 
Human Resources (NSF) Special Emphasis Panel 
on Evaluation. 

The Special Emphasis Panel on Evaluation 
provided technical assistance and 
recommendations on the full range of activities 
supported under the evaluation program of the 
NSF Education and Human Resources Directorate 
(EHR). 

EHR’s highly visible programs needed the 
support of ongoing formative evaluation for their 
continuous improvement. In addition, many 
external audiences, including the United States 
Congress, wanted to know what EHR’s programs 
were accomplishing in relationship to their stated 
purposes. The Division of Research, Evaluation, 
and Communication (REC) provides leadership 
for EHR program evaluation. 

Joint Accomplishments of the Evaluation and 
Policy Analysis of Systemic Reform Teams 

Research on Equity 

A joint accomplishment of the Systemic Reform 
Teams is a set of papers, discussions, and a forum 
concerning equity in student achievement. Two 
research monographs were written, by William 
Tate (1997, mathematics) and Alberto Rodriguez 
(1997, science), exploring differences in student 
achievement on a variety of measures and data 
sets disaggregated by race, ethnicity, and gender. 
The papers showed various kinds of differences 
and trends across genders and ethnic groups — 
European, Asian, African, Latino, and other. One 
finding, for example, was of small differences 
between the genders on tests of science, but a 
substantial difference in favor of males on the 
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mathematics portion of the SAT. This 
mathematics/gender difference remained 
constant across ethnic groups, which 
themselves differed substantially from each 
other in average scores. Another finding was 
mixed evidence for the proposition that 
similar course taking will produce similar 
results. Such course taking narrows but does 
not eliminate achievement gaps and does not 
occur with sufficient frequency to allow 
comparisons for many students. Discussion of 
these papers was far ranging and indicated a 
possible need for new insights and approaches 
toward equity. However, the value of 
longitudinal data on outcomes disaggregated 
by ethnicity, gender, and subject matter was 
evident. Systemic reforms that do not collect 
and analyze such data are ignoring an 
important tool. 

Jane Butler Kahle, who served as NISE 
Fellow on the Policy Team, prepared a 
research monograph (1998b) and an NISE 
Brief (1998a) on reaching equity in systemic 
reform. Both the brief and the monograph 
provide analysis of educational equity in 
science and mathematics and, using a research 
paradigm, build a practical model that is 
applicable for districts, schools, and 
classrooms around the country. Key 
indicators, applicable across many types of 
educational systems, are enrollment and 
achievement in eighth-grade mathematics, 
quality of the content and instruction of 
science and mathematics courses, narrowing 
of achievement gaps among subgroups of 
students, and changes in teaching practices to 
align them with the mathematics and science 
standards. Sharon Lynch, who also served as 
NISE Fellow, published a book on Equity and 
Science Education Reform (2000) in which 
she uses both research-based studies and 
classroom experiences of teachers and 
students to analyze the complexities of equity 
concerns across ethnicity, class, gender, and 



location. Okhee Lee prepared an NISE research 
monograph on equity in content standards (Lee, 
1998), which resulted in a Review of Educational 
Research publication 
(1999). 

The 2000 NISE Forum on Equity Issues in 
Mathematics and Science covered several 
important aspects of equity, including what we 
know about effective methods to address equity 
issues and what research is needed. Panelists 
represented a wide range of expertise in equity 
issues from across the nation. Discussion and 
networking opportunities aided the Forum in 
achieving its overall goal: to draw together leaders 
in the field and stimulate intellectually rich 
conversations to develop a better understanding of 
diversity and equity issues in mathematics and 
science education. In the papers, presentations, 
and small group sessions, participants stressed the 
importance of learning how to make diversity an 
asset rather than a problem to overcome by 
thinking of students as individuals, not as broad 
categories, by training teachers to be sensitive to 
equity and diversity issues, by using strong 
models of disseminating information, by aligning 
curriculum, instruction, and assessment, and by 
conducting cross-cultural research. A synthesis of 
the think pieces from the discussion sessions is 
available (Britton, Raizen, Kaser, & Porter, 2000). 

The Study of Systemic Reform in Milwaukee 
Public Schools (SSR-MPS) 

In April 1998, the Joyce Foundation agreed to 
fund this project for an initial period of two years 
at about $1 million per year. Additional funding 
for this project was obtained from the Helen 
Bader Foundation, Inc., of Milwaukee. A second 
two-year Joyce grant has been awarded. 

Clune and Webb codirect this project. The 
ultimate goal of the research is understanding how 
systemic policy can fulfill its promise as a 
powerful method of sustaining widespread school 
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improvement and major gains in student 
achievement over the long run in the 
Milwaukee Public Schools (MPS). To reach 
that goal, the project has three principal 
purposes: 

1 . generate useful knowledge and 
recommendations for policy in the district; 

2. allow impartial observers, funding agencies, 
and system managers to understand the system 
and its performance at a deeper level; and 

3. impart analytical capacity to the district so 
that SSR-MPS can be phased out, or assume a 
reduced role, after a period of years. 

Research is being conducted in six areas: 
policy and governance; alignment of 
curriculum standards, student assessments, 
and classroom instruction; capacity of data 
systems and analytic expertise to deliver 
relevant information about student and system 
performance; school improvement; 
professional development; and students with 
exceptional educational needs. 

Research is currently being conducted in the 
projects listed below. 

• Technical assistance and quantitative 
analysis on implementing the value-added 
system of standardized testing currently 
under consideration by the School Board. 

• Technical assistance and analytical 
research on how individual schools in 
Milwaukee can best acquire data from the 
district data system, and analyze these and 
other data, to best meet the needs of 
schools. 

• Research on the extent to which the middle 
school proficiencies have been successful 
as a means of raising standards for all 
students, focusing professional 



development, and stimulating school 
improvement. 

• Research and technical assistance on how the 
proposed system of embedded classroom 
assessments in high school can be optimally 
successful in meeting district and school 
goals. 

This project will both build capacity in the district 
and produce written reports and publications for a 
national research audience. 

Over the first 2+ years of funding, the project 
made great progress. 

- Focused the entire project more closely on 
student achievement, acquired a new project 
liaison at a higher level in the district (Deputy 
Superintendent Jacqueline Patterson), and 
obtained a letter of endorsement for the work of 
the project from the new Superintendent. 

- Became a regular and influential participant in 
advising the Department of Research and 
Assessment and other departments on the new 
system of assessment and accountability currently 
being designed and implemented in the district. 

- Placed a team member on the district’s data 
warehouse development team. His charge is to 
work on translating district-level and school-level 
analytical models for the Team. 

- Worked with two middle schools on how to 
report achievement data for accountability. 

- Worked with the Audit Office of the Board of 
Governors on how to analyze and report adequate 
school performance. 

- Held a one-day workshop with MPS leadership 
and consultants from Minneapolis on assessment 
and accountability. 
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- Produced and presented the first value-added 
analysis of student achievement scores in 
MPS. 

- Gained access to and/or set up on our 
computers longitudinal and other data on 
student achievement in middle schools in four 
subjects. 

- Obtained middle school proficiency data for 
all students in spring 2000 and consulted on 
the database and reporting for these data. 

The project research staff also presented a 
symposium at the annual American 
Educational Research Association meeting in 
April 2000. The session was entitled 
Interdisciplinary Action Research on Systemic 
Reform in an Urban District. The session 
featured the following presentations and 
related papers: 

Embedded Research on Systemic Reform and 
the Design Experiment: Similarities and 
Dissimilarities by William H. Clune 

Embedded Research in Practice: A Study of 
Systemic Reform in Milwaukee Public Schools 
by Norman L. Webb 

Looking for School Improvement in 
Milwaukee: An Example of Embedded 
Research by Jennifer O’ Day 

Including All Students in Accountability 
Assessment by Jeffery P. Braden 

Value-added Measures of Student 
Achievement by Robert Meyer 

III. Staff, Fellows, and Collaborators 

Staff: William Clune and Norman Webb 
(Team Leaders), Jeffery Braden, Janice 



Downer, Ron Jetty, Sarah Mason, Robert Meyer, 
Susan Millar, Jennifer O’Day, Eric Osthoff, Andy 
Porter, Senta Raizen, Alberto Rodriguez, William 
Tate, Christopher Thom, Paula White, and John 
Wright. 

Fellows: Charles Bruckerhoff, Hugh Burkhardt, 
Judith Burry-Stock, Jeanne Rose Century, Kathy 
Comfort, Norma Davila, Andrew Jackman, Jane 
Butler Kahle, Okhee Lee, Paul LeMahieu, Sharon 
Lynch, James Ridgway, and Uri Treisman. 

Collaborators: Christopher Anderson, Steven 
Bauman, Vicki Bier, Thomas Carpenter, Donald 
Chambers, Deborah Tepper Haimo, Michael 
Kirst, Michael Knapp, Judy Roitman, Tom 
Romberg, Pat Rossman, Richard Rossmiller, 
Bassam Shakhashiri, and John Witte. 

NISE Publications of the Systemic Reform 
Team 

Britton, E., Raizen, S., Kaser, J., & Porter, A. 
(2000). Beyond description of the problems: 
Directions for research on diversity and equity 
issues in K-12 mathematics and science 
education. Washington, DC: National Center for 
Improving Science Education. 

Clune, W. H. (1999). Toward a theory of systemic 
reform: The case of nine NSF statewide systemic 
initiatives (Research Monograph No. 16). 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Clune, W. H., Haimo, D. T., Roitman, J., 
Romberg, T., Wright, J. C., & Wright, C. S. 
(1997). Commentaries on mathematics and 
science standards (Occasional Paper No. 3). 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Clune, W. H., Millar, S. B., Raizen, S. A., Webb, 
N. L., Bowcock, D. C., Britton, E. D., Gunter, R. 
L., & Mesquita, R. (1997). Research on systemic 
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reform: What have we learned? What do we 
need to know? Synthesis of the Second Annual 
NISE Forum (Workshop Report No. 4, Vol. 

1). Madison: University of Wisconsin-* 
Madison, National Institute for Science 
Education. 

Clune, W. H., Millar, S. B., Raizen, S. A., 
Webb, N. L., Bowcock, D. C., Britton, E. D., 
Gunter, R. L., & Mesquita, R. (1997). 

Research on systemic reform: What have we 
learned? What do we need to know? Synthesis 
of the Second Annual NISE Forum (Workshop 
Report No. 4, Vol. 2). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

Heck, D. J., & Webb, N. L. (2000). Purposes - 
and issues of systemic evaluation in education 
(Research Monograph No. 20). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

House, E. R. (1997). Implementing evaluation 
findings in government agencies (Occasional 
Paper No. 2). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

Kahle, J. B. (1998a). Measuring progress 
toward equity in science and mathematics 
education (NISE Brief Vol. 2, No. 3). 

Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Kahle, J. B. (1998b). Reaching equity in 
systemic reform: How do we assess progress 
and problems (Research Monograph No. 9). 
Madison: University of Wisconsin-*Madison, 
National Institute for Science Education. 

Kirst, M. S., & Bird, R. L. (1997). The politics 
of developing and maintaining mathematics 
and science curriculum content standards 
(Research Monograph No. 2). Madison: 



University of Wisconsin-Madison, National 
Institute for Science Education. 

Kirst, M., Bird, R., & Raizen, S. A. (1997). 
Tensions between mathematics and science 
content standards and local politics (Occasional 
Paper No. 5). Madison: University of Wisconsin- 
Madison, National Institute for Science 
Education. 

Knapp, M. S. (1997). Between systemic reforms 
and the mathematics and science classroom: The 
dynamics of innovation implementation, and 
professional learning (Research Monograph No. 
1). Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Lee, O. (1998). Current conceptions of science 
achievement in major reform documents and 
implications for equity and assessment (Research 
Monograph No. 12). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

Raizen, S. A. (1997). Standards for science 
education (Occasional Paper No. 1). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

Ridgway, J. (1998a). From barrier to lever: 
Revising roles for assessment in mathematics 
education (NISE Brief Vol. 2, No. 1). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

Ridgway, J. (1998b). The modeling of systems and 
macro-systemic change: Lessons for evaluation 
from epidemiology and ecology (Research 
Monograph No. 8). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

Rodriguez, A. J. (1997). Counting the runners 
who dont have shoes: Trends in student 
achievement in science by socioeconomic status 
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and gender within ethnic groups (Research 
Monograph No. 3). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

Rodriguez, A. J. (1999). Equity through 
systemic reform: The case of whole-school 
mathematics and science restructuring in 
Puerto Rico (Research Monograph No. 14). 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Rodriguez, A. J. (2000). A cross-case analysis 
of the Puerto Rico Statewide and the Miami- 
Dade Urban Systemic Initiatives: Promising 
examples of equity in systemic reform 
(Research Monograph No. 21). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

Tate, W. (1997). Race, SES, gender, and 
language proficiency trends in mathematics 
achievement: An update (Research 
Monograph No. 4). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

Watson, J., & Zeyher, S. (2000). Education 
system information needs and systemic reform 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 
Manuscript in preparation. 

Webb, N. L. (1997a). Criteria for alignment of 
expectations and assessments in mathematics 
and science education (Research Monograph 
No. 6). Madison: University of Wisconsin- 
Madison, National Institute for Science 
Education. 

Webb, N. L. (1997b, January). Determining 
alignment of expectations and assessments in 
mathematics and science education (NISE 
Brief Vol. 1, No. 2). Madison: University of 



Wisconsin-Madison, National Institute for 
Science Education. 

Webb, N. L. (1999a). Alignment of science and 
mathematics standards and assessments in four 
states (Research Monograph No. 18). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

Webb, N. L. (1999b). Evaluation of systemic 
reform in mathematics and science: Synthesis 
and proceedings of the Fourth Annual NISE 
Forum (Workshop Report No. 8). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

Webb, N. L. (1999c). State A: Alignment between 
standards and assessments in science for grades 3 
and 8 and mathematics for grades 3 and 6 
(Project paper for Council of Chief State School 
Officers and National Institute for Science 
Education). Madison: University of Wisconsin, 
National Institute for Science Education. 

Webb, N. L. (1999d). State B: Alignment between 
standards and assessments in mathematics for 
grades 4, 8, and 10 (Project paper for Council of 
Chief State School Officers and National Institute 
for Science Education). Madison: University of 
Wisconsin, National Institute for Science 
Education. 

Webb, N. L. (1999e). State C: Alignment between 
standards and assessments in science for grades 4 
and 8 and mathematics for grades 4 and 8 
(Project paper for Council of Chief State School 
Officers and National Institute for Science 
Education). Madison: University of Wisconsin, 
National Institute for Science Education. 

Webb, N. L. (1999f). State D: Alignment between 
standards and assessments in science for grades 
3, 7 and 10 and mathematics for grades 4 and 8 
(Project paper for Council of Chief State School 
Officers and National Institute for Science 
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Education). Madison: University of 
Wisconsin, National Institute for Science 
Education. 

Webb, N. L., & Heck, D. J. (1996). 
Evaluation strategies project: A report of the 
evaluation questions conference of the 
evaluation strategies project (Workshop 
Report No. 2). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

NISE-Related Publications of the Systemic 
Reform Team 

Boone, W. J., & Kahle, J. B. (1998). Student 
perceptions of instruction, peer interest, and 
adult support for middle school science: 
Differences by race and gender. Journal of 
Women and Minorities in Science and 
Engineering, 4{A), 333-340. 

Century, J. R. (2000). Capacity. In N. L. 
Webb, J. R. Century, N. Davila, D. Heck, & 
E. Osthoff (Eds.), Evaluation of systemic 
reform in mathematics and science 
(Wisconsin Center for Education Research, 
University of Wisconsin-Madison). 
Manuscript in preparation. 

Century, J. R. (2000). Equity. In N. L. Webb, 
J. R. Century, N. Davila, D. Heck, & E. 
Osthoff (Eds.), Evaluation of systemic reform 
in mathematics and science. (Wisconsin 
Center for Education Research, University of 
Wisconsin-Madison). Manuscript in 
preparation. 

Century, J. R. (2000). Role of evaluator. In N. 
L. Webb, J. R. Century, N. Davila, D. Heck, 
& E. Osthoff (Eds.), Evaluation of systemic 
reform in mathematics and science. 
(Wisconsin Center for Education Research, 
University of Wisconsin-Madison). 
Manuscript in preparation. 



Century, J. R. (2000). Sustainability. In N. L. 
Webb, J. R. Century, N. Davila, D. Heck, & E. 
Osthoff (Eds.), Evaluation of systemic reform in 
mathematics and science. (Wisconsin Center for 
Education Research, University of Wisconsin- 
Madison). Manuscript in preparation. 

Clune, W. H. (1998). The national standards in 
math and science: Developing 
consensus, unresolved issues, and unfinished 
business. Teachers College Record, 700(1), 6-7. 

Clune, W. H. (1998, Fall). The “standards wars” 
in perspective. Teachers College Record, 700(1), 
144-149. 



Clune, W. H., Porter, A. C., & Raizen, S. A. 

(1999, September 29). Systemic reform: What is 
it? How do we know? Education Week, 79(5), 31. 

Davila, N. (2000). Design of evaluation of 
systemic reform. In N. L. Webb, J. R. Century, N. 
Davila, D. Heck, & E. Osthoff (Eds.), Evaluation 
of systemic reform in mathematics and science. 
(Wisconsin Center for Education Research, 
University of Wisconsin-Madison). Manuscript in 
preparation. 

Davila, N. (2000). Instmction. In N. L. Webb, J. 

R. Century, N. Davila, D. Heck, & E. Osthoff 
(Eds.), Evaluation of systemic reform in 
mathematics and science. (Wisconsin Center for 
Education Research, University of Wisconsin- 
Madison). Manuscript in preparation. 

Davila, N. (2000). Linkages. In N. L. Webb, J. R. 
Century, N. Davila, D. Heck, & E. Osthoff (Eds.), 
Evaluation of systemic reform in mathematics and 
science. (Wisconsin Center for Education 



Clune, W. H., Osthoff, E. J., & White, P. A. 
(2000). Theory and practice of systemic reform of 
mathematics and science education. (Wisconsin 
Center for Education Research, University of 
Wisconsin-Madison). Manuscript in preparation. 



Research, University of Wisconsin-Madison). 
Manuscript in preparation. 

Gamoran, A., Porter, A. C., Smithson, J., & 
White, P. A. (1997). Upgrading high school 
mathematics instruction: Improving learning 
opportunities for low-achieving, low-income 
youth. Educational Evaluation and Policy 
Analysis. 19(4), 325-338. 

Haimo, D. T. (1998, Fall). Are the NCTM 
standards suitable for systemic adoption? 
Teachers College Record, 700(1), 45-65. 

Heck, D. J. (1998). Evaluating equity in 
statewide systemic initiatives: Asking the right 
questions. Journal of Women and Minorities 
in Science and Engineering, 4(2,3), 161-181 . 

Heck, D. (2000). Purposes and questions for 
evaluation of systemic reform. In N. L. Webb, 
J. R. Century, N. Davila, D. Heck, & E. 
Osthoff (Eds.), Evaluation of systemic reform 
in mathematics and science. (Wisconsin 
Center for Education Research, University of 
Wisconsin-Madison). Manuscript in 
preparation. 

Heck, D. (2000). Saturation. In N. L. Webb, 

J. R. Century, N. Davila, D. Heck, & E. 
Osthoff, (Eds.), Evaluation of systemic reform 
in mathematics and science. (Wisconsin 
Center for Education Research, University of 
Wisconsin-Madison). Manuscript in 
preparation. 

Kahle, J. B. (1997). Systemic reform: 
Challenges and changes. Science Educator, 
6 ( 1 ), 1 - 6 . 

Kahle, J. B. (1998). Equitable systemic reform 
in science and mathematics: Assessing 
progress. Journal of Women and Minorities in 
Science and Engineering, 4(2,3), 91-112. 



Kahle, J. B., & Rogg, S. R. (1997). Assessing 
systemic change: Ohio’s Statewide Systemic 
Initiative, Discovery. Systemic Initiatives, 7(3), 10- 
1 1. Available: http://www.ehr.nsf.gov/ehr/esr 
/sysinit/news.htm 

Kirst, M. W., & Bird, R. L. (1997). The politics of 
developing and sustaining mathematics and 
science curriculum content standards. Advances in 
Educational Administration, 5, 107-132. 

Knapp, M. S. (1997, Summer). Between systemic 
reforms and the mathematics and science 
classroom: The dynamics of innovation, 
implementation, and professional learning. 

Review of Educational Research, 67(2), 227-266. 

Lee, O. (1999). Equity implications based on the 
conceptions of science achievement in major 
reform documents. Review of Educational 
Research, 69(1), 83-115. 

Lee, O., & Fradd, S. H. (1988). Science for all, 
including students from non-English language 
backgrounds. Educational Researcher, 27(4), 12- 
21 . 

Lee, O., & Paik, S. (2000). Conceptions of 
science achievement in major reform documents. 
School Science and Mathematics, 700(1), 16-26. 

Lynch, S. (2000). Equity and science education 
reform. Mahway, NJ: Lawrence Erlbaum. 

Osthoff, E. (2000). Incentives. In N. L. Webb, J. 

R. Century, N. Davila, D. Heck, & E. Osthoff 
(Eds.), Evaluation of systemic reform in 
mathematics and science. (Wisconsin Center for 
Education Research, University of Wisconsin- 
Madison). Manuscript in preparation. 

Paik, S., & Lee, O. (1998). Analysis of the 
conceptions of science achievement in major 
reform documents in the United States and Korea. 
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Journal of the Korean Association for 
Research in Science Educationy 7S(4), 571- 
587. 

Paik, S., & Lee, O. (in press). Science 
achievement: Synthesis of current conceptions 
in major reform documents in the United 
States and Korea. Elementary Science 
Education (South Korea). 

Porter, A. C. (1994). National standards and 
school improvement in the 1990s: Issues and 
promise. American Journal of Education^ 
702(4), 421-449. 

Porter, A. C. (1995). The uses and misuses of 
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Researcher^ 24(1), 21-27. 

Porter, A. C. (1998a). Dilemmas in assessing 
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education (pp. 339-350). Washington, DC: 
American Psychological Association. 

Porter, A. C. (1998b). The effects of 
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123-172). Washington, DC: Brookings 
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College Level One 



I. Team Mission 

The objective of the NISE College Level One 
(CL-1) Team is to enhance the quality of 
introductory postsecondary SMET courses, 
reflecting their role as curriculum “pressure 
points” in shaping student career trajectories, 
influencing science literacy, and promoting 
equity. The National Science Foundation 
(NSF) published the report Shaping the 
Future: New Expectations for Undergraduate 
Education in Science, Mathematics, 
Engineering, and Technology that summarized 
practices and innovative approaches for 
teaching postsecondary SMET courses and 
provided recommendations that could lead to 
improved outcomes in these courses. The 
Shaping the Future report helped the CL-1 
Team align its program with NSF and 
stakeholder aspirations for CL-1 SMET 
courses. Our efforts focused on conducting 
research that identifies and validates effective 
practices in these courses and on 
disseminating this information to key 
stakeholder audiences. We identified and 
validated practices for dissemination through 
annual CL-1 Institutes. Each CL-1 Institute 
focused on a topic of critical importance to 
college SMET instruction to bridge the large 
gulf between the knowledge resulting from 
education research and the knowledge base 
and practice of college SMET instructors. The 
topical expertise was generated by a select 
group of NISE Fellows, representing both 
SMET instructors and education researchers; 
the CL-1 Team provided the ongoing 
infrastructure. Major themes addressed by the 
CL-1 Institutes were cooperative learning, 
classroom assessment of student learning, and 
the appropriate use of technology in student 
learning. Each CL-1 Institute synthesized 
what is known in higher education SMET 
research, prepared products based on this 



synthesis, including Web sites (all available at 
http ://www. wcer. wise, edu/n is e/cl 7 ) , and 
disseminated these products to the college SMET 
instructional community. The ultimate goal of the 
CL-1 Team has been to develop a comprehensive, 
integrated Web-based resource of effective 
teaching practices. The CL-1 Innovations in 
SMET Education Web site provides a launch pad 
for reform-ready instructors, a support base for 
campus and national professional development 
centers and workshop providers, and a reference 
site for the education and research directorates of 
the National Science Foundation. 

II. Accomplishments 

Institute on Collaborative Learning 

In consultation with colleagues at the National 
Science Foundation, the CL-1 team investigated 
the scholarly basis for the use of cooperative 
learning in introductory college SMET courses. A 
meta-analysis convincingly demonstrated that 
using small group learning methods in SMET 
courses, rather than traditional methods, produce 
significant gains in student achievement, 
persistence, and attitude. The effect size of 0.5 
that was found is highly significant. This study, 
“Effects of Small-Group Learning on 
Undergraduates in Science, Mathematics, 
Engineering and Technology: A Meta-Analysis,” 
was published in Review of Educational Research 
(Springer, Stanne, & Donovan, 1999) and 
represented the first such analysis spanning 
SMET disciplines and the spectrum of 
postsecondary institutions. A summary article of 
this research for chemistry audiences was 
published in the Journal of Chemical Education 
(Bowen, 2000). 

Given the importance of the findings from the 
meta-analysis, the CL-1 Team implemented an 
aggressive strategy for bringing collaborative 
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learning methods to the attention of college 
SMET instructors, advisors, and 
administrators through the development of a 
Web site. The heart of the Collaborative 
Learning Web site, developed under the 
leadership of Anthony Jacob and Ann 
Burgess, is an instructor-friendly tutorial on 
collaborative learning techniques. In addition, 
the site provides personal stories and advice 
from SMET instructors who have adopted this 
pedagogical approach; the research foundation 
for collaborative learning (including the meta- 
analysis report); an annotated bibliography on 
collaborative learning that is available through 
NISE; a perspective article in Journal of 
College Science Teaching, “Small-Group 
Instruction in Science, Mathematics, 
Engineering and Technology: A Discipline 
Status Report and a Teaching Agenda for the 
Future” (Cooper & Robinson, 1998); and a 
compilation of frequently asked questions. 
Constructive suggestions that improved the 
site were obtained from the Communicating 
with Mass Audiences (CMA) Team and from 
national experts who attended a workshop 
hosted by CL-1 . To advertise the Web site, 
Kate Fahl developed a brochure that has been 
distributed to thousands of college SMET 
instructors and administrators through 
disciplinary professional meetings and 
collaborations with organizations such as 
Project Kaleidoscope. 

Institute on Assessment of Student Learning 

As college SMET instructors begin to 
experiment with new forms of pedagogy and 
content, the question inevitably arises about 
whether their students are learning more as the 
changes are implemented. A CL-1 Institute 
was developed to explore issues associated 
with the classroom assessment of student 
learning. An NISE Forum and a book 
developed by an NISE Fellow helped to 
launch this effort, which resulted in the 



development of a Web site, the Field-tested 
Learning Assessment Guide (FLAG). 

The theme of the 1998 NISE Forum was 
“Indicators of Success in Postsecondary SMET 
Education: Shapes of the Future.” The Forum 
engaged a broad cross-section of college SMET 
stakeholders in a dialog regarding the importance 
of assessment. A proceedings volume was 
prepared by Susan Millar and is available from 
NISE. As background scholarship for the 
Institute’s work, a book on assessment research. 
Assessing Science Understanding. A Human 
Constructivist View, was co-edited by NISE 
Fellow Joel Mintzes and published by Academic 
Press. The volume presents critical analyses from 
national experts on various assessment tools. 

Part of the vision underpinning the Institute on 
Assessment of Student Learning was that the 
Institute’s products would help reform-ready 
practitioners to think of assessment beyond the 
traditional awarding of grades. Many methods of 
assessment — including portfolios, concept maps, 
and interviews — are being introduced into college 
SMET courses to better characterize student 
learning. A practitioner-oriented resource 
developed by the Institute is the Field-tested 
Learning Assessment Guide (FLAG) Web site, 
which can be accessed through the above URL. 
The content of the site comprises a primer that 
provides a rationale for integrating assessment 
with college SMET courses and introduces 
common terms associated with assessment; a 
facilitator that helps college SMET instructors 
identify appropriate goals and align them with 
assessment methods; and a collection of so-called 
Classroom Assessment Techniques, or CATs, and 
associated tools. The CATs were commissioned 
from nationally recognized leaders in the 
assessment of student learning and include 
contributions from the NSF-funded Math 
Assessment Resource Service (MARS) project. 
All contributions were vetted through an Editorial 
Board drawn from national experts. Like the 
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Collaborative Learning Web site, the FLAG 
Web site was created to serve as a core 
resource for the CL-1 SMET instructional 
community. Through a successor NSF grant to 
former NISE Fellow Michael Zeilik, one of 
the original developers of the Web site, the 
FLAG will continue to be updated and 
expanded, as befits a moving target like 
assessment. The NISE Formative Evaluation 
Team’s initial evaluation of the FLAG site 
indicates that it is meeting its objectives of 
serving the college SMET instructional 
community as a central assessment resource. 
As was done for the cooperative learning 
project, an aggressive dissemination effort 
was implemented, including preparation of a 
widely distributed brochure and presentations 
at high-profile conferences, including 
meetings of the American Educational 
Research Association (AERA) meeting and 
the American Association of Higher 
Education (AAHE). 

Institute on Learning Technology 

The introduction of technology into college 
SMET courses is changing the landscape 
associated with these courses. New 
technologies include computers, sensors, and 
multimedia presentation tools. Collectively, 
such technologies are reshaping college 
SMET education by providing opportunities 
for distance learning, access to enormous 
databases, and remote use of state-of-the-art 
instrumentation, to cite Just a few examples. 
The CL-1 Institute on Teaching with 
Technology was established to explore the 
impact of technology and to help college 
SMET instructors identify appropriate uses of 
technology. A Technology Education 
workshop served as a springboard for this 
effort, and the Institute has developed a Web 
site that is intended to serve as a resource for 
the college SMET teaching community. 



A three-day workshop was held at NSF 
headquarters during the summer of 1999. The 
workshop brought together leading scholars in 
education and cognitive research and college 
instructors who are pioneers in the use of 
technology in SMET at the college level. The 
workshop report (TechEd 99; 
http://www.wcer.wisc.edu/teched99/) resulting 
from this meeting identified key technical, 
pedagogical, political and economic issues 
associated with the introduction of technology. 

With support from the NSF CISE Directorate’s 
National Partnership for Advanced Computing 
Infrastructure (NPACI), the 1999-00 CL-1 
Institute on Learning Technology studied how 
SMET faculty are successfully using new 
technologies in their classrooms and laboratories. 
Under the leadership of NISE Fellow Susan 
Millar, the CL-1 Institute was to provide college 
SMET instructors with better resources for 
understanding why and how they should use 
technology-enhanced learning strategies. The 
Team (a) conducted ten case studies of exemplary 
technology use in a variety of SMET disciplines 
and higher education environments; (b) developed 
a set of twenty “vignettes,” illustrating a range of 
technology-enhanced practices in disciplines and 
types of institutions not represented in the case 
studies; and (c) gathered from a diverse sample of 
200 SMET faculty responses to frequently asked 
questions about the use of technology in college 
SMET courses. Analysis of all these materials 
revealed key factors for successful innovation and 
adaptation of technology in support of student 
learning. A comprehensive taxonomy of learning 
technologies that informs the case study analysis 
and helps shape future research in learning 
technology was also developed. The materials 
resulting from these studies have been 
incorporated into another CL-1 Web site. In 
addition, the findings will be developed into 
workshop materials for distribution by 
professional organizations. 
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III. Staff and Collaborators 

Staff: Arthur B. Ellis (Team Leader), Robert 
D. Mathieu (Director, CL-1 Institute), Briana 
Bright, Gina Brissenden, Aaron Brower, Ann 
Burgess, Susan Daffinrud, Samuel Donovan, 
Jack Husted, Anthony Jacob, Kate Loftus 
Fahl, Walter Secada, and Leonard Springer. 

Fellows: Jean-Pierre Bayard, Lia Brillhart, 
Craig Bowen, Steve Ehrmann, John Jungck, 
Eileen Lewis, Flora McMartin, Joel Mintzes, 
Marco Molinaro, Elaine Seymour, Sheila 
Tobias, and Michael Zeilik. 

Collaborators: Clifford Adelman, James 
Cooper, Patricia Cross, Douglas Duncan, 
Roscoe Giles, Mary Kennedy, Louise Liao, 
Catherine Middlecamp, Susan Millar, 

Terrence Millar, John Moore, Greg Moses, 
Jeanne Narum, James Ridgway, Pamela 
Robinson, Gloria Rogers, Karl Smith, and 
Mary Beth Stanne. 

Products 

Collaborative Learning Web site: 

http://www.wcer.wisc.edu/nise/cll/CL/default. 

asp 

College Level One Web site: 
http://www. wcer. wise. edu/nise/CLl 

Field-tested Learning Assessment Guide 
(FLAG) Web site: 

http://www.wcer.wisc.edu/nise/cll/flag/default 

.asp 

Learning Through Technology Web site: 
http://www. wcer.wisc.edu/nise/cll/ilt/default. 
asp 

Student Assessment of their Learning Gains 
(SALG) Web site: 

http://www.wcer.wisc.edu/salgains/instructor/ 



Collaborative Learning brochure (available 
through NISE central office) 

Field-tested Learning Assessment Guide brochure 
(available through NISE central office) 

Learning Through Technology brochure 
(available through NISE central office) 

Student Assessment of their Learning Gains 
brochure (available through NISE central office) 

NISE Publications of the College Level One 
Team 

Cooper, J., & Robinson, P. (1998). Small-group 
instruction: An annotated bibliography of science, 
mathematics, engineering and technology 
resources in higher education (Occasional Paper 
No. 6). Madison: University of Wisconsin- 
Madison, National Institute for Science 
Education. 

Millar, S. B. (Ed.). (1998). Indicators of success 
in postsecondary SMET education: Shapes of the 
future. Synthesis and proceedings of the Third 
Annual NISE Forum (Workshop Report No. 6). 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Springer, L., Stanne, M. E., & Donovan, S. S. 
(1998). Effects of small- group learning on 
undergraduates in science, mathematics, 
engineering, and technology: A meta-analysis 
(Research Monograph No. 1 1). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

NISE-Related Publications of the College Level 
One Team 

Adelman, C. (1997). Leading, concurrent, or 
lagging ? The knowledge content of computer 
science in higher education and the labor market 
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(ED Publication No. PLLI 97-8046). 
Washington, DC: U.S. Department of 
Education. 

Adelman, C. (1998). Women and men of the 
engineering path: A model for analysis of 
undergraduate careers (ED Publication No. 
PLLI 98-8055).. Washington, DC: U.S. 
Department of Education. 

Bowen, C. W. (2000). A quantitative literature 
review of cooperative learning effects on 
high-school and college chemistry 
achievement. Journal of Chemical Education, 
77(1), 116-119. 

Committee on Undergraduate Science 
Education, National Research Council. (1999). 
Transforming undergraduate education in 
science, mathematics, engineering, and 
technology. Washington, DC: National 
Academy Press. 

Cooper, J., & Robinson, P. (1998). Small 
group instruction in science, mathematics, 
engineering, and technology: A discipline 
status report and a teaching agenda for the 
future. Journal of College Science Teaching, 
27(6), 383-388. 

Mathieu, R. (2000). Assessment tools to drive 
learning: FLAG, SALG, and other proven 
assessments available online. In Targeting 
curricular change: Reform in undergraduate 
education in science, math, engineering, and 
technology (pp. 26-31). Washington, DC: 
American Association for Higher Education. 

Millar, S. (2000). Opening doors to new 
spaces: The processes of implementing 
education reform. In Targeting curricular 
change: Reform in undergraduate education 
in science, math, engineering, and technology 
(pp. 41-45). Washington, DC: American 
Association for Higher Education. 



Mintzes, J. J., Wandersee, J. H., & Novak, J. D. 
(Eds.). (1998). Teaching science for 
understanding: A human constructivist view. San 
Diego, CA: Academic Press. 

Mintzes, J. J., Wandersee, J. H., & Novak, J. D. 
(Eds.). (2000). Assessing science understanding: 

A human constructivist view. San Diego, CA: 
Academic Press. 

Springer, L. (1998). Research on cooperative 
learning in college science, mathematics, 
engineering, and technology. Cooperative 
Learning and College Teaching, S(3), 2-4. 

Springer, L., Stanne, M. E., & Donovan, S. S. 
(1999). Effects of small-group learning on 
undergraduates in science, mathematics, 
engineering, and technology: A meta-analysis. 
Review of Educational Research, 69(1), 21-51. 

Tobias, S. (1999). Some recent developments in 
teacher education in mathematics and science. A 
review and commentary. Journal of Science 
Education and Technology, S(l), 21-31. 

Wiese, D., Seymour, E., & Hunter, A. (1999, 
June). Report on a panel testing of the student 
assessment of their learning gains instrument by 
faculty using modular methods to teach 
undergraduate chemistry. Boulder: University of 
Colorado, Bureau of Sociological Research. 

Wright, J. C., & Wright, C. S. (1998). A 
commentary on the profound changes envisioned 
by the national science standards. Teachers 
College Record, 700(1), 122-143. 

Wright, J. C., Millar, S. B., Kosciuk, S. A., 
Penberthy, D. L., Williams, P. H., & Wampold, B. 
E. (1998). A novel strategy for assessing the 
effects of curriculum reform on student 
competence. Journal of Chemical Education, 75, 
986-992. 
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Professional Development 



I. Team Mission 

The NISE Professional Development Team 
dedicated itself to identifying what constitutes 
effective professional development for science 
and mathematics teachers and spreading that 
knowledge to education leaders working to 
improve science and mathematics teaching in 
K-12 classrooms. Through its five-year 
program of work the Team has had a 
significant impact on the professional 
development practices in many reform 
initiatives, school districts, and state 
departments of education. It has helped staff 
developers throughout the nation to think 
more critically about their own practices. 

The questions that drove the Team’s work 
were: 

• What is currently known about science 
and mathematics, learning, teaching, 
professional development, and the change 
process that can improve the design and 
effectiveness of learning opportunities for 
teachers? 

• How should the specific goals of 
professional development for a particular 
set of teachers and the context in which 
teacher learning opportunities are being 
planned influence the design of 
professional development? 

• What alternative strategies and approaches 
to professional learning exist, how are they 
used, and how effective are they in 
assisting teachers to learn what they need 
to know to effect student learning of 
science and mathematics? 

• What practices help to bridge the gap 
between preservice education and teacher 
professional development? 

• What skills and knowledge do professional 
developers need in order to provide high 
quality learning experiences for teachers? 



II. Accomplishments 

The Team’s accomplishments in addressing each 
of the seven goals are outlined below. 

Goal 1: To document the characteristics of 
effective professional development for science 
and mathematics teachers and develop a 
framework to guide the redesign of 
professional learning opportunities for 
teachers. 

• Developed and disseminated NISE Brief, 
Principles of Effective Professional 
Development for Mathematics and Science 
Education (Loucks-Horsley et al.), a synthesis 
of the professional development standards 
produced by the National Council of Teachers 
of Mathematics (NCTM), the National 
Research Council (NRC), the National Staff 
Development Council (NSDC), and the 
National Center for Improving Science 
Education. 

• Created the Professional Development Design 
Framework. In collaboration with NISE 
fellows Hubert Dyasi, Susan Friel, Judy 
Mumme, Cary Sneider, and Karen Worth and 
advisors Josefina Arce, Joan Ferrini-Mundy, 
Deborah Schifter, Vernon Sells, Mark St. 

John, and Iris Weiss, the Team developed and 
documented the process through which 
effective professional development is 
designed. The goal of developing a framework 
for the design of professional learning 
opportunities for K-12 inservice science and 
mathematics teachers evolved during the first 
year as a direct result of work with the NISE 
fellows. The intent was to capture the learning 
of current professional development efforts to 
increase the knowledge base from both “craft 
wisdom” and disciplined inquiry and make 
that information accessible to practitioners and 
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researchers alike. At team meetings, 
professional developers shared their work 
by describing how they had developed 
their programs and by explaining the 
critical decisions they had made that 
contributed to the particular forms their 
programs had taken. The outcome of these 
deliberations was a consensus that 
effective professional development 
practice could best be characterized by 
means of a Design Framework. The 
decisions that are made in designing 
professional learning opportunities for 
science and mathematics teachers are the 
product of a cyclical design process that is 
informed and influenced by various inputs. 
The Team identified the inputs to 
professional development design that must 
be considered, including (1) contextual 
factors such as existing policies, teacher 
knowledge and skills, curriculum, student 
achievement levels; (2) the knowledge 
base on teaching, learning, the nature of 
science and mathematics, professional 
development, and the change process; (3) 
critical issues such as how the project will 
reach scale, how it will develop 
professional culture, address equity, and 
evaluate its results; and (4) the 
professional learning strategy or 
combination of strategies (e.g., coaching, 
study groups, workshops) that will best 
address the professional development 
needs and purpose. 

• Wrote and disseminated a book on the 
Professional Development Design 
Framework. The Team summarized 
findings and recommendations in the book 
Designing Effective Professional 
Development for Teachers of Science and 
Mathematics, published by Corwin Press 
in 1998. Over 1 1,000 copies have been 
sold. Each of the key sections of the 
framework is elaborated with illustrations 



and vignettes, often drawn from the work of 
the NISE Fellows. The audiences for the book 
are, first, those in positions to design and 
conduct professional development and, 
second, those who fund, consume, conduct 
research on, evaluate, and develop policies for 
professional development. 

• Collaborated with the Eisenhower National 
Clearinghouse to disseminate findings more 
broadly. Summarized the Design Framework 
and the 15 major professional development 
strategies in two publications — one for science 
educators and one for mathematics educators. 
Over 50,000 of these books have been 
disseminated throughout the nation. 

Goal 2: To create a dialogue among 
professional developers and other education 
leaders for elaborating and understanding 
what constitutes effective professional 
development and for promoting the use of the 
NISE Design Framework and other Hndings. 

• Engaged with leaders in professional 
development. The project engaged with 
professional development leaders around the 
NISE findings through annual and regional 
professional meetings and ongoing email 
communication. The Team conducted sessions 
at conferences of NCTM, the National Science 
Teachers Association (NSTA), and NSDC 
each year and invited participants to try out 
the findings and recommendations of the 
project and communicate with staff about their 
results. Many participants maintained contact 
through email and follow-up at other 
meetings. The Team also conducted learning 
sessions for NSF systemic initiatives and 
teacher enhancement projects, the Glenn 
Commission, WestEd’s National Academy for 
Science and Mathematics Education 
Leadership, large school districts, and many 
regional and state service providers. Through 
these efforts over 3,000 leaders of professional 




33 



38 



development became aware of the NISE 
Professional Development Design 
Framework and ideas for improving 
learning for science and mathematics 
teachers. The Professional Development 
Team has also interacted with preservice 
educators and professional developers 
from around the country in an ongoing 
dialogue about what is known about 
effective development of science and 
mathematics teachers and the critical 
issues that surround professional 
development. 

• Conducted First Annual NISE Forum, 
1996. The Forum’s primary purposes were 
to (1) extend and strengthen understanding 

- of practices and issues in professional 
development, with implications for design 
of more effective programs and initiatives 
in mathematics and science education, and 
(2) build a community of those seeking 
ongoing interaction around professional 
development practices and issues. A 
secondary purpose was to design a 
conference that modeled many of the 
principles of effective professional 
development. The Forum included a 
plenary presentation of the Design 
Framework, discussions of eight case 
studies of professional development in 
science and mathematics, and discussions 
of critical issues that cut across all 
professional development efforts, e.g., 
equity, leadership, scaling up. 

Goal 3: To develop case examples of the 

NISE Professional Development Design 

Framework in action. 

• Developed four case studies of 
professional development practice. The 
Team studied the ways in which the 
Design Framework and its central 
components are implemented in different 



contexts and the organizations, structures, and 
methods that serve to facilitate their effective 
implementation and dissemination. The case 
studies were committed to using the NISE 
Design Framework to analyze the results of 
completed efforts or plan ongoing efforts and 
documenting and analyzing their 
implementation efforts. 

Edward Silver, Margaret Smith, and Mary 
Kay Stein wrote retrospective case studies of 
two sites that were part of the QUASAR 
Project, a multiyear project funded by the 
Ford Foundation (1990-1995) that focused on 
the teaching of mathematics in middle schools 
in large urban school districts. Cases 
examined the professional development 
efforts at these two sites, using the design 
framework as a tool for describing and 
analyzing the professional development 
decisions that were made. The case studies 
drew on existing data about teachers, resource 
partners from local universities, and the 
professional development experiences 
collected over the five years of the project. 
The cases detail the curricular decisions made 
(one site implemented a reform-oriented 
mathematics curriculum developed through 
funding from the NSF; the other site used a 
curriculum to be created by the teachers and 
unified by the concepts of ratio and 
proportion) and the professional development 
activities used to support these curricula. 
These cases are valuable in helping 
professional developers understand the 
importance of key factors such as the 
mentoring by teachers of their new colleagues 
while they adopted the school’s innovative 
curriculum and relationships between teachers 
and resource partners from local universities. 

NISE staff Susan Mundry and fellow Ned 
Levine wrote a case study about a staff 
development project funded by the NSF as a 
Rural Systemic Initiative project involving 42 
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schools in 22 school districts in 
southwestern Montana. The project goal 
was to implement new strategies in 
science and technology education in these 
schools through a variety of professional 
development strategies such as teacher 
institutes, leadership training, mentor 
teacher training, and direct classroom 
support. The case considered three 
schools: a small school (about 300 
students), a one-room school, and a school 
serving predominantly Native American 
students. It focused on the question of the 
integrity of the curriculum as it was 
implemented across distance and in 
different contexts of culture and local 
control. The case is valuable in providing 
professional developers with insights into 
the difficulties imposed on effective 
professional development by distance, 
isolation, and cultural context. 

Team member Kathy Stiles wrote a case 
study about science and mathematics 
professional development in a large urban 
school district that is culturally diverse. In 
the district, professional development in 
science and mathematics was designed and 
coordinated by six people who formed a 
team that worked with the district Staff 
Development office to ensure that district 
and program goals were met. For science 
and mathematics, curricular goals included 
application of science and mathematics in 
the real world, competence in the use of 
technology, and a diversification of 
teaching strategies. Professional 
development activities were designed to 
increase teacher skills and confidence in 
the use of reform-oriented teaching 
strategies and to facilitate 
interdisciplinary, thematic instruction. The 
case describes the history of professional 
development planning, implementation in 
the district, and the intervention of Stiles’ 



application of the Design Framework with the 
Team. 

• Published case study findings. The Team and 
collaborators conducted a cross-case analysis 
and identified a number of dilemmas faced by 
professional developers as they planned and 
carried out their efforts. These were described 
in three publications — in the Harvard 
Educational Review, The High School 
Principal Magazine, and the NISE Brief, (See 
product list.) 

• Presented cases at 1998 AERA annual 
meeting 

Goal 4: To investigate and document 
conditions and/or programs that bridge the 
gap between preservice and inservice 
education. 

• Conducted a literature review and prepared a 
summary paper on the relationship between 
the preservice and inservice components of 
science and mathematics teachers’ 
professional development and the principles 
that should guide the development of this 
relationship in the future. 

• Convened collaborators with expertise in 
particular approaches to connecting pre- and 
inservice teacher education (e.g., in 
professional development schools, in novice 
teacher mentor and induction programs, in 
teacher assessment). The collaborators were 
chosen because they had implemented 
programs or approaches to bridge the 
preservice-inservice gap, brought different 
perspectives, e.g., from colleges, schools, or 
state education agencies, and had disciplinary 
backgrounds in mathematics and/or science. 
They wrote a detailed statement of the 
problem and the factors that contribute to the 
disconnect between preservice and inservice 
and generated recommendations for 
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developing stronger, more coherent 
relationships between schools and 
colleges. 

• NISE Fellow Barbara Spector, with Susan 
Mundry, Katherine Stiles, and Susan 
Loucks-Horsley, documented initiatives 
and conditions that build connections 
between preservice and inservice 
education and summarized them in a 
compendium and report prepared as NISE 
Research Monograph No. 17. (See product 
list.) 

Goal 5: To deepen understanding of three 
different professional development 
strategies, their impact in different 
' contexts, with different teacher learning 
goals, and in different combinations. 

• Prepared a literature review on three 
specific strategies for professional 
development — curriculum 
implementation, immersion in inquiry and 
problem solving, and case discussions — 
entitled “Strategies for Professional 
Development: What Do We Know and 
Still Need to Know?” 

• Documented three specific professional 
development strategies that build 
pedagogical content knowledge in science 
and mathematics. The Team recruited 
NISE collaborators to investigate three 
strategies — curriculum implementation, 
immersion in inquiry and problem solving, 
and case discussions — more deeply. The 
staff, collaborators, and team fellow 
Cheryl Brown-Kovacic convened three 
times to explore the purpose and outcomes 
of these strategies and to demonstrate how 
teachers engage in them. Collaborators 
wrote papers that describe an intervention 
using the different strategies. Staff and 
Brown-Kovacic wrote an introduction 



chapter and worked closely with the 
collaborators to develop a coherent set of 
cases on the three professional development 
strategies. 

Goal 6: To study the effects of knowledge 
utilization strategies to improve professional 
development practice. 

• Established and convened a cadre of 
professional developers who are using the 
project’s research to make improvements in 
professional learning experiences for teachers 
of science and mathematics. The Professional 
Development Team documented how the 
cadre uses knowledge from NISE research to 
make improvements in teacher learning. 

• Drafted guidebook of learning activities for 
preparing professional developers to design 
and provide quality teacher learning 
experiences. 

• Designed and conducted professional learning 
activities. The Professional Development team 
and PD cadre developed and tried out over 40 
learning activities designed to increase 
professional developers knowledge of 
effective professional development and 
professional development design. Activities 
that were successful in meeting the desired 
outcomes are included in the guidebook draft. 

III. Staff, Fellows, and Collaborators 

Staff: Susan Loucks-Horsley (Team Leader), 
Peter Hewson, Nancy Love, Susan Mundry, 
Katherine Stiles, Kathy Dunne, and Mary 
Stenson. 

Fellows: Cheryl Brown-Kovacic, Hubert Dyasi, 
Susan Friel, Uwe Hilgert, Ned Levine, Judith 
Mumme, Margaret Schwan Smith, Edward Silver, 
Cary Sneider, Barbara Spector, Mary Kay Stein, 
and Karen Worth. 
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Collaborators: Josefina Arce, Came Barnett, 
Virginia Bastable, Cathy Carroll, Karen 
Cerwin, Kathy DiRanna, Mark Driscoll, 

Kathy Dunne, Joan Ferrini-Mundy, David 
Hartney, Gregg Humphrey, Barbara Miller, 
Jean Moon, Jan Phlegar, Lynn Rankin, Ann 
Rosebery, Susan Jo Russell, Deborah Schifter 
Vernon Sells, Jerome Shaw, Mark St. John, 
Rita Starnes, Jo Topps, and Iris Weiss. 

NISE Publications of the Professional 
Development Team 

Hewson, P., & Loucks-Horsley, S. (1997). 
Professional development for science and 
mathematics education: Putting knowledge 
into action. A synopsis of the First Annual 
Forum of the National Institute for Science 
Education (Workshop Report No. 3). 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Loucks-Horsley, S., Hewson, P., Love, N., & 
Stiles, K. (1998). Designing professional 
development for teachers of science and 
mathematics. Thousand Oaks, CA: Corwin 
Press. 

Loucks-Horsley, S., Stiles, K., & Hewson, P. 
(1996). Principles of effective professional 
development for mathematics and science 
education: A synthesis of standards (NISE 
Brief Vol. l,No. 1). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

Mundry, S., & Loucks-Horsley, S. (1999). 
Designing professional development for 
science and mathematics teachers: Decisions 
points and dilemmas (NISE Brief Vol. 2, No. 

1 ). Madison: University of Wisconsin- 
Madison, National Institute for Science 
Education. 
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Mundry, S., Spector, B., Stiles, K., Loucks- 
Horsley, S. (1999). Working toward a continuum 
of professional learning experiences for teachers 
of science and mathematics (Research Monograph 
No. 17). Madison: University of Wisconsin- 
Madison, National Institute for Science 
Education. 

NISE-Related Publications of the Professional 
Development Team 

Eisenhower National Clearinghouse. (1998). 

Ideas that work: Mathematics professional 
development. Columbus, OH: Author. (Digest 
of mathematics portion of Loucks-Horsley, 
Hewson, Love, & Stiles, 1997) 

Eisenhower National Clearinghouse. (1999). 

Ideas that work: Science professional 
development. Columbus, OH: Author. (Digest of 
science portion of Loucks-Horsley, Hewson, 

Love, & Stiles, 1997) 

Horsley, D. L., & Loucks-Horsley, S. (1998). 
CBAM brings order to the tornado of change. 
Journal of Staff Development, 79(4), 17-20. 

Loucks-Horsley, S. (1998, Spring). The role of 
teaching and learning in systemic reform: A focus 
on professional development. Science Educator, 
7(1), 1-6. 

Loucks-Horsley, S. (1998, Summer). JSD fomm: 

I have changed my emphasis. Journal of Staff 
Development, 79(3), 7-8. 

Loucks-Horsley, S., & Bybee, R.W. (1998). 
Implementing the National Science Education 
Standards. The Science Teacher, 65(6), 22-25. 

Mundry, S., Spector, B., Loucks-Horsley, S., & 
Morrison, N. (1999, December 1). From novice to 
pro: Improving science and mathematics teaching 
by improving teacher learning. Education Week, 
79(14), 21. 
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Mundry, S., Spector, B., Loucks-Horsley, S., 
& Morrison, N. (2000, January-February). 
From novice to pro: Improving science and 
mathematics teaching by improving teacher 
learning. NSTA Reports, 77(4), 53. 

Stein, M. K., & Mundry, S. (1999, October). 
Professional development for science and 
mathematics teachers: Dilemmas of design. 
The High School Principal Magazine, 7(2), 
14-19. 



Stein, M. K., Smith, M. S., & Silver, E. A. (1999). 
The development of professional developers: 
Learning to assist teachers in new settings in new 
ways. Harvard Educational Review, 69(3), 237- 
269. 

Stiles, K. E., & Loucks-Horsley, S. (1998). 
Professional development strategies. The Science 
Teacher, 65(6), 46-49. 
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Secondary Teacher Education Project 



I. Team Mission 

Reforming teacher education is a current 
national priority (NRC, 1996). This continuing 
project is about promoting standards-based 
reform in teacher education through Web- 
based instructional design. An important 
question addressed through our work is how to 
effectively build Web-based instruction that 
helps teachers acquire truly useful knowledge 
that supports them at work and in other real- 
world contexts. Such knowledge encompasses 
not only pedagogical and subject-matter 
concepts and skills that underlie teaching, but 
also the tools of lifelong, self-directed 
professional development. 

The phrase useful knowledge emphasizes an 
ambitious goal for knowledge transfer, that 
knowledge acquired in one context (e.g., 
school, Web sites) be recalled and used in 
another context (e.g., the workplace). 
Knowledge transfer has been an elusive goal 
for education; researchers, instructional 
developers, and knowledgeable educators 
eagerly seek ways to obtain it. Studies by 
cognitive psychologists and anthropologists 
have repeatedly shown that nontrivial 
knowledge transfer is difficult to achieve (e.g., 
Salomon & Perkins, 1989). 

The premise is that high level transfer of 
professional knowledge and skills, including 
the skills of reflective practice and lifelong 
learning, can be attained through Web-based 
instructional design. By Web-based 
instructional design we refer to both design of 
Web environments and their communication 
tools, and the social structures of their use. 
Little is known about how to engineer 
effective instructional Web environments. 

This work will lead to general design solutions 
and principles that may apply to many 



professional fields involving application of 
complex domain knowledge to real-world practice 
(e.g., medical education). 

II. Accomplishments 

Development of STEP Web Prototype 

Our work involved design and implementation of 
the STEP Web {http://www.wcer.wisc.edu/step), a 
learning environment on the World Wide Web for 
supporting scientific instruction for teacher 
education. The phrase scientific instruction refers 
to both the design of the Web site and its subject 
matter content; both are based on current 
principles of learning and development drawn 
from cognitive psychology and other learning 
sciences. STEP Web and its associated 
instructional approaches were tried and evaluated 
in teacher education programs at UW -Madison 
and Rutgers University in the 1999-2000 spring 
semester. The field trials were successful; an 
upgraded version of STEP Web is currently being 
used and evaluated in the UW-Madison teacher 
education program. In the following we describe 
the STEP site, the theoretical rationale for its 
design, and its use in teacher education. We then 
summarize the results of our field trials and other 
research. 

The STEP Web goals are to help future teachers 
acquire pedagogically relevant structural 
knowledge about the disciplinary concepts they 
are teaching; scientific knowledge about how 
students learn and develop within disciplines; and 
the ability to combine, adapt, and apply this 
knowledge base to analyze and improve teaching 
and learning. 

A theory about how to design instruction that will 
insure transfer of learning to professional life is 
Cognitive Flexibility Theory (CFT; e.g., Spiro, 
Feltovich, Jacobson, & Coulson, 1991). 
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Appropriate domains for the application of 
CFT include professional-level education, 
such as medical and teacher education. A 
central argument of CFT is that many 
instructional approaches fail because they 
represent complex subject matter in an 
unrealistically simplified and well-structured 
manner. The most common kind of failure 
observed is reductive bias — the tendency to 
oversimplify approaches and solutions to 
complex problems. The remedy for learning 
deficiencies related to domain complexity 
requires instructional designs that afford 
greater cognitive flexibility. 

This includes the ability to represent 
knowledge from different conceptual 
and case perspectives and then, when 
knowledge must later be used, the 
ability to construct from those different 
conceptual and case representations a 
knowledge ensemble tailored to the 
needs of the understanding or problem- 
solving situation at hand. (Spiro et al., 
1991, p. 24) 

A decade ago the developers of CFT 
suggested that instruction with 
multidimensional, nonlinear hypertext systems 
has the power to convey knowledge 
complexity and promote features of cognitive 
flexibility. Today this observation applies to 
site design on the World Wide Web. A CFT 
approach to instructional Web design proceeds 
as follows; (1) A problem-solving domain is 
defined (such as application of subject-matter 
knowledge and scientific learning principles to 
teaching); (2) a library of cases representing 
real-world problems and solutions is 
constructed (such as video-based stories about 
actual classroom lessons); (3) a library of 
resources for understanding and analyzing the 
cases is developed (such as Web pages about 
learning theories and educational research); 
and (4) a system of connections is created to 



permit navigation through the Knowledge Web 
(such as Web links among video cases and case- 
relevant learning research and theories). The 
resulting Knowledge Web represents a 
realistically complicated, nonlinear network of 
conceptually complex relationships among cases 
and ideas within a domain. 

Accordingly, STEP Web is a highly 
interconnected network of pages that interlink the 
following types of resources in a conceptually 
meaningful manner: 

• cases — stories of student learning and 
development resulting from lessons in 
actual classrooms — that include edited 
video of the classroom plus additional 

' materials that supply information about 
context 

• instructional problems and projects that 
make use of cases and are designed to 
promote in-depth analysis for development 
of knowledge about how to support 
student learning 

• a network of case-related links to Web 
pages and other resources discussing core 
concepts from cognitive psychology and 
other learning sciences that teachers need 
to know 

• access to expert case analyses 

• links to online discussions and human 
expertise 

• links to additional tools and resources that 
teachers can use to help them adapt and 
implement ideas acquired from study of 
cases 

Implementation and Evaluation of STEP Web 

In the 2000 spring semester, 55 teacher education 
students were assigned to small groups of 5-7 
students that studied together within a Problem 
Based Learning (PBL) format. During the 
semester, each student participated in two 
different PBL groups. Each group was assigned a 



40 




45 



problem and a case to study. For example, a 
case assigned to a group of science majors 
was “Students Get a Charge out of Static 
Electricity.” This case, presented on STEP 
Web as readings, videos, and inquiry 
materials, tells the story of an actual science 
unit in a public school taught by a popular 
teacher and representing a good case of 
traditional instruction. The problem was to 
redesign the unit and justify that redesign in 
psychological terms. 

The work of the students was guided in class 
by a tutor — a TA who received training in 
guiding PEL. After studying the case 
individually on STEP Web, students began 
their group work in class by discussing the 
case and identifying learning issues — things 
they needed to learn more about to solve the 
redesign problem. Between classes, students 
researched their learning issues, bringing 
findings to their group discussions. STEP Web 
was made available as an optional research 
tool that could be used outside of class. The 
links and navigational tools in STEP Web 
guided students’ research; by exploring paths 
from links, students were able to obtain 
information about case-relevant concepts and 
other resources and pursue them to a desired 
depth. Research beyond the materials in STEP 
Web was also promoted, since links led to 
other library and Web resources. Some 
students also purchased and used optional 
textbooks. 

The problem required about four weeks to 
complete. A tutor guided students through 
class discussions of their research, during 
which time they identified positive and 
negative aspects of the instruction within the 
case and proposed new instructional solutions. 
In the third week they posted their redesign 
with explanations on a Web conference site 
for peer evaluation and consultation with 
experts, including scientists and educational 



experts. After revision, a group report was 
submitted as a course requirement. 

The STEP implementation at UW-Madison during 
the 2000 spring semester represented a process of 
continuous user-centered design in which students 
provided feedback that was used to upgrade and 
improve STEP Web throughout the semester. 

Early in the semester, intensive feedback was 
obtained from a small number of students who 
volunteered to be research subjects, but on March 
7 and again on April 18, all students were 
surveyed to obtain their feedback and satisfaction 
ratings regarding the Web site. 

Fifty-four of the 55 students returned surveys on 
March 7; 50 returned surveys on April 18. On 
these dates, 48 and 46 students respectively 
reported using the Web site as an instructional 
resource for their study and PEL research. 
Satisfaction with STEP Web as an instructional 
resource was 3.9 on March 7 and 4.1 on April 18, 
based on a rating scale of 1-5 (not very satisfied 
to very satisfied). Students’ comments initiated a 
number of improvements and changes throughout 
the semester. For example, the addition of a 
search engine was based on students’ requests. 
Students’ satisfaction with STEP Web increased 
as the site was improved and students gained 
experience. For example, one student who 
participated in three surveys commented: 

• Feb 22: “KW (Knowledge Web) - 

impressed me this week. ... I did not 
research outside of it.” (No rating 
requested) 

• Mar 7: “I am getting better at navigating 

the KW.” (Rating = 4) 

• Apr 18: “I am starting to appreciate the 

Knowledge Web.” (Rating = 5) 

Other representative student comments: 

• “When I finally figured out how to use it, it 

was great.” (Rating = 5) 
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• “I like the newer KW.” (Rating = 4) 

• “Much improved!” (Rating = 4) 

• “Some pages that could have helped 

weren’t up.” (Rating = 4) 

• “They [Web pages] were quite useful, 

but KW needs to be more easily 

navigated.” (Rating = 3) 

• “I found the KW to be confusing in 

some of its explanations.” (Rating = 3) 

In sum, most students in the UW-Madison 
course were pleased with the Knowledge Web 
by midsemester, but their comments indicated 
that further development and improvement is 
needed. Based on students’ concerns, there is 
need to (1) add to and improve resources on 
STEP Web, (2) improve navigation, and (3) 
provide instructional supports within the 
course to speed the process of learning how to 
use the site. We are currently working on 
these improvements. 

STEP Web was also used at Rutgers 
University in a smaller course taught by an 
experienced PBL instructor. There it was 
positively rated, despite being at an early stage 
in its development. Based on fourteen students 
and a scale of 1-5 (not very satisfied to very 
satisfied), the Web site was rated 4.6. The 
textbook used in the course, a best-selling 
educational psychology text, was rated 4.5. 

The evaluation study at UW-Madison 
produced statistical evidence of significant 
growth in students’ ability and propensity to 
activate and combine concepts from the 
learning sciences in the analysis of videotaped 
lessons. In addition, a microgenetic analysis of 
a single PBL group, composed of science 
education majors, demonstrated that 
discussions of cases provided opportunity for 
learners to socially construct meanings, share 
knowledge, negotiate conflicts, and integrate 
ideas from other teacher education courses 
and experiences. Significant shifts in 



individuals’ conceptual understandings of student 
learning and development were documented. 

These findings suggest that the STEP instructional 
model as a whole is viable in terms of course 
goals. 

Development of Video Cases for STEP 

Recent work suggests that teachers improve their 
instruction when presented opportunities to view 
and discuss cases of classroom practice (e.g., 
Merseth, 1996; Shulman, 1992). Facilitated case 
discussions are deemed an especially promising 
method for teacher education, since preservice 
teachers often do not have control of classrooms 
in which to try out their teaching ideas (Putnam & 
Borko, 2000). 

Publications from the Third International 
Mathematics and Science Study (TIMSS; e.g., 
Stigler & Hiebert, 1999), coupled with release of 
videotapes of 235 eighth-grade mathematics 
lessons {http://timssvideo.psych.ucla.edu) from 
three countries, has increased enthusiasm for 
using video cases of classroom practice as a basis 
for case discussions. Many researchers are now 
designing and investigating various strategies, 
technologies, and social structures for helping 
make video case discussions an aspect of teaching 
culture (e.g., Barab et al., 2000; Frederiksen et al., 
1999; Lampert & Ball, 1998; Marx, Blumenfeld, 
Krajcik, & Soloway, 1998). For example, Barab et 
al. developed a video-based Internet technology 
that enables teachers to upload video of their 
classes and remain in their classrooms while they 
go online to observe and discuss how teachers in 
other sites are implementing state standards. Marx 
et al. (1998) describe a multimedia system called 
CAPPS (Casebook of Project Practices) that 
scaffolds teachers as they study and learn from 
video cases of classroom practice. Stigler 
(personal communication) and Goldman-Segal 
(personal communication) are developing Internet 
technologies to support teachers’ group 
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discussions and learning from instructional 
video cases. 

Unlike most approaches mentioned above, 
STEP requires Internet video cases that are 
carefully selected, edited, and designed to 
teach a particular body of curricular 
knowledge (Derry, Kim, Steinkuehler, Street, 
Canty, Fassnacht, Hewson, Hmelo, & Spiro, 
2000). Cases include many components: 
edited video of classrooms, expert analyses, 
teacher commentary, and supplementary 
materials. We have developed six instructional 
video cases and associated instructional 
materials. These cases represent lessons in 
middle and high school science (2 cases); high 
school mathematics; high school foreign 
language; high school English; and high 
school social studies. More cases are currently 
under development, and we are seeking funds 
to continue and expand this work. 

Web Tools for Supporting Distributed 
(distance forms) of PBL 

Building tools to support a distributed form of 
PBL is necessary due to the widespread 
problem of resources for undergraduate 
education. By putting STEP PBL activities 
online, we will avert many physical, temporal, 
and human resource constraints that face 
STEP and greatly increase the potential 
number of students served. Our strategy 
involves adding features to STEP Web that 
will better prepare each individual group 
member to use the STEP Web site and to 
participate in group work. Also, most group 
discussions will be removed from the 
classroom to an asynchronous online 
environment. This approach will permit 
facilitated PBL groups for distance courses 
and will have a number of other advantages 
over our current STEP implementation, which 
requires that each group meet face-to-face in 
class each week with a trained tutor. 



We have made significant progress in building 
and testing a new conferencing tool that supports 
distributed (distance) PBL in STEP. This addition 
will also provide new users with early guidance in 
use of STEP Web and will therefore minimize 
some reported difficulties in learning to use the 
site effectively. 
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Dissemination Programs 

Interacting with Professional Audiences 



I. Team Mission 

The IPA team reaches out to diverse 
professional audiences to encourage use of 
NISE products and research results, obtain 
reactions to them, and elicit suggestions for 
future research directions. The IPA Team does 
this directly through organizing the annual 
NISE Fomms, but also assists other NISE 
teams in planning and conducting 
dissemination activities. Each Forum 
convenes representatives of the many NISE 
target audiences to share research and 
developments in an area of science and 
mathematics education reform selected for its 
timeliness and salience. The IPA Team also 
has recmited and is working with some 25 
selected professional organizations that are 
collaborating with the NISE more intensively. 
These collaborating organizations assist in the 
work of NISE and in disseminating NISE 
results in ways that are tailored for reaching 
the organizations' members. 

Specifically, the IPA team’s goals are 

• to share knowledge and information 
generated by the NISE with key 
stakeholders; 

• to promote interaction and dialogue about 
the knowledge and information generated 
by the NISE among key stakeholders; and 

• to encourage policymakers and 
practitioners to use research knowledge to 
improve science, mathematics, 
engineering, and technology education. 



II. Accomplishments 

The Annual NISE Forum 

The five annual NISE Forums have been very 
successful in four ways: (1) creating awareness 
and a generally favorable perception of NISE and 
its work, (2) generating state-of-the-art knowledge 
on pressing topics in the SMET community, (3) 
helping shape NISE’s and individual teams’ 
research agendas, and (4) developing an effective 
process for all participants to contribute to the 
substance and results of large-scale SMET 
education meetings. Throughout the five years, 
the NISE Forum has been fully subscribed, 
drawing on a rich mix of some 300 participants at 
each Forum; it also has attracted supplemental 
corporate support each year. Individuals from 
virtually every U.S. state have represented many 
kinds of institutions (dozens of professional 
organizations, all types of postsecondary 
institutions, a variety of research and development 
projects, and diverse types of school districts). 
Participants' feedback from the formal evaluation 
questionnaire as well as anecdotal comments have 
been overwhelmingly positive. The feedback from 
each Forum has been used to shape the succeeding 
one. 

The caliber of Forum participants has made it very 
worthwhile to collect and analyze the perspectives 
offered in the Forum's small-group discussions 
and include them in the Forum reports. The 
experts who have been enlisted to speak also have 
contributed to the reports through their prepared 
papers that are circulated in advance of the Forum. 
The continuing value of the Forums is 
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evidenced by the publications resulting from 
them or having been partially shaped by the 
panel papers and discussions. For example, 
the Year 1 Forum discussions led to the 
commercially published book by Susan 
Loucks-Horsley and her PD team. Teachers 
College Press has expressed interest in 
monographs authored by the Systemic Reform 
Team, one on policy analysis and one on 
evaluation of systemic reform initiatives, with 
inputs from the Forums held in Years 2 and 4. 
The Year 3 Forum on assessment issues in 
postsecondary education helped launch the 
Web site FLAG (Field-tested Learning 
Assessment Guide), an interactive site posting 
assessment examples designed and used by 
SMET faculty. The Year 5 Forum on equity 
issues in SMET has proved important enough 
that the U.S. Department of Education 
supported the writing and production of a 
summary report, to be widely distributed. 

Some papers written by the Forum speakers 
have independently been receiving acclaim. 
Susan Loucks-Horsley’s Year 1 paper was 
reproduced in Science Educator, the 
newsletter of the National Science Educational 
Leadership Association. Many requests for 
Uri Treisman's Year 2 paper led to the 
Institute plans to transform it into an NISE 
Brief. The National Research Council 
circulated several papers from the Year 3 
Forum to participants in one of its conferences 
on undergraduate education. 

The effectiveness of the Forum processes to 
generate and disseminate knowledge is 
evidenced by the fact that NSF requested the 
NISE IPA Team’s assistance in conducting 
two other similar conferences in SMET 
education, one on teacher preparation and one 
on graduate education. Moreover, a guide to 
designing effective meetings and conferences, 
based on the NISE development and 
experience with the Forums, was prepared by 



the IPA staff and has been published by Corwin 
Press. 

The Forum is a good example of how the Institute 
has successfully drawn together the work of its 
teams. For the first four Forums, though the IPA 
Team coordinated all aspects of the Forum, the 
cognizant research team was responsible for the 
substantive content and agendas. The research 
teams have worked hard not only to highlight their 
NISE work but also to draw in leading experts 
from other organizations. The Year 5 Forum was 
an all-institute effort, since equity issues cut 
across all the teams’ research. The NISE 
management regularly monitored and facilitated 
the development of each year’s Forum. The 
Formative Evaluation (FE) team has helped 
design the details of the Forum evaluations as well 
as analysis of small-group discussions to feed into 
the published reports. 

Collaborating with Other Organizations 

The IPA Team initiated relationships with some 
25 organizations interested in collaborating with 
NISE. As a result, several organizations have 
formally lent their names and/or efforts to putting 
on one of the annual NISE Forums: American 
Association of Higher Education, Consortium for 
Policy Research in Education, Council of Chief 
State School Officers, National Council of 
Teachers of Mathematics (NCTM), National 
Governors’ Association, National Staff 
Development Council (NSDC), and National 
Science Teachers Association (NSTA). Also, the 
publication divisions of NCTM and NSDC 
formally codistributed the PD Team's book, and 
the Eisenhower National Clearing House 
produced two spin-off publications based on the 
book, one in mathematics and one in science. 
Senior staff of several of the collaborating 
organizations have provided critical reviews of 
draft NISE publications and attended working 
meetings of NISE research teams. Based on their 
knowledge of NISE work, NCTM, NSDC, and 
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(particularly) NSTA have invited NISE staff 
to make featured presentations at their annual 
and/or regional conferences, and several 
organizations have asked NISE staff to confer 
with them on issues of mutual interest. 

Dissemination Assistance 

The most notable success of the IPA Team’s 
efforts to work with NISE staff on 
dissemination activities is its work to 
negotiate and execute the commercial 
publication of the PD team's work by Corwin 
Press and the secondary distribution of the 
book by NCTM and NSDC. This book is 
making far more impact than if it had been 
self-published by NISE. For example, in its 
first month, the book already had become one 
of Corwin's top three all-time sellers. As noted 
earlier, the Institute looks forward to two 
further externally published monographs, 
initially facilitated by the IPA team. Also, 
Education Week accepted two articles 
submitted by the Team (see NISE-Related 
Publications). 
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1998 Special NISE Forum. “Strengthening 
Graduate Education in Science and Engineering: 
Promising Practices and Strategies for 
Implementation” 

1999 Fourth Annual NISE Forum. “Evaluation of 
Systemic Reform in Mathematics and Science” 

1999 Special NISE Forum. “From Preparation to 
Practice: NSF Teacher Education PI Workshop” 

2000 Fifth Annual NISE Forum. “Research on 
Equity Issues in SMET Education” (Report to be 
published and distributed in fall 2000.) 

NISE Insider, quarterly electronic newsletter to 
NISE Collaborating Organizations 

1997-1999 annual meetings of NISE 
Collaborating Organizations 
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N. L., Bowcock, D. C., Britton, E. D., Gunter, R. 
L., & Mesquita, R. (1997). Research on systemic 
reform: What have we learned? What do we need 
to know? Synthesis of the Second Annual NISE 
Forum (Workshop Report No. 4, Vol. 1). 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Clune, W. H., Millar, S. B., Raizen, S. A., Webb, 
N. L., Bowcock, D. C., Britton, E. D., Gunter, R. 
L., & Mesquita, R. (1997). Research on systemic 
reform: What have we learned? What do we need 
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to know? Synthesis of the Second Annual NISE 
Forum (Workshop Report No. 4, Vol. 2). 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Webb, N. L. (1999). Evaluation of systemic 
reform in mathematics and science. Synthesis 
and proceedings of the Fourth Annual NISE 
Forum (Workshop Report No. 8). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

White, P. A. (1998). NISE Fellows program: 
Feedback from past Fellows (Occasional 
Paper No. 7). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 



NISE-Related Publications of the Interacting 
with Professional Audiences Team 

Clune, W. H., Porter, A. C., & Raizen, S. A. 
(1999, September 29). Systemic reform: What is 
it? How do we know? Education Week, ?9(5), 31. 

Mundry, S., Britton, E., Raizen, S., & Loucks- 
Horsley, S. (2000). Designing successful 
professional meetings and conferences in 
education: Planning, implementation, and 
evaluation. Thousand Oaks, CA: Corwin Press. 

Mundry, S., Spector, B., Loucks-Horsley, S., & 
Morrison, N. (1999, December 1). From novice to 
pro: Improving science and mathematics teaching 
by improving teacher learning. Education Week, 
79(14), 21. 
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Communicating with Mass Audiences 



I. Team Mission 

The general objectives of the CMA Team have 
been to promote public understanding of SMET- 
related concepts and issues and to understand how 
the new communication channel of the World 
Wide Web can be used most effectively to 
communicate SMET concepts to mass audiences. 
During the first three years, the CMA Team’s 
primary effort was designing and producing The 
Why Files {http://whyfiles.news.wisc.edu), the 
award winning World Wide Web science 
magazine created by NISE. At the end of Year 3, 
funding for The Why Files site was shifted from 
NISE to the Graduate School of the University of 
Wisconsin-Madison. At that time, the focus of the 
team became conducting research on the 
communication of science via the World Wide 
Web, an endeavor the team began in Year 2. 

Production Component 

While traditional mass media such as newspapers, 
television, and radio play a critical role in 
informing the public about news and issues of 
science and technology, they are often limited in 
their resources and ability to provide depth and 
context. And while traditional media pique public 
curiosity about science and technology. The Why 
Files seeks to satisfy that interest by offering in- 
depth, informed, and critical looks at current 
topics such as Lyme disease, planets discovered 
beyond our solar system, and genetic engineering. 

Within the last two years, the media landscape has 
been altered as new science news venues have 
emerged on the World Wide Web. While a few of 
these sites seek to capitalize on popular interest in 
science, many cater to specific constituencies. For 
instance, there are a number of science news 
venues that provide news and information for 
audiences connected in some way to the life 



sciences. However, these and most other 
general science news sites tend to recycle print 
news or to simply post news releases from 
research organizations. Curiously, there are 
very few venues developing purely online 
science content such as that produced by The 
Why Files. 

Research Component 

Beginning in Year 2, the CMA team 
conducted evaluative research on The Why 
Files specifically and more general research 
on how science learning takes place on the 
World Wide Web. This latter work became 
the sole focus of the team beginning in Year 4, 
resulting in a change in the CMA team 
leadership. The research plans for the 4-year 
period were to (1) understand the users of 
science information on the Web (e.g., 
demographics, motivations), particularly users 
of The Why Files; (2) understand how 
individuals use science information on the 
Web, particularly their navigation patterns 
through Web sites; and finally, (3) understand 
the effects of Web presentations of science 
information on information processing and 
learning. Thus, our research program began 
with a descriptive focus during Years 2 and 3 
and moved into a more explanatory phase for 
Years 4 and 5. 

II. Accomplishments to Date 

The Production Component 

The Why Files continues to expand its unique 
content base, producing between 45 and 50 
new illustrated feature articles per year. In the 
past year. The Why Files has covered an array 
of topics such as moving species from the 
endangered list and the discovery of marine 
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worms that live to be an astounding 250 years of 
age. Other packages covered the battle to teach 
evolution in Kansas, the world’s malaria epidemic 
and the technology that underpins a proposed 
missile defense system. The Why Files production 
team is now engaged in a site redesign exercise to 
improve navigation and accessibility for Why 
Files readers. A number of those changes have 
been driven directly by the research conducted on 
The Why Files site by members of the CMA 
Team. 

Changes in package design and approach to a 
given topic may be subtle or pronounced, but the 
basic formula that has made The Why Files a 
popular success remains. That formula involves a 
carefully crafted combination of accurate but 
humorously written text, eye-catching graphics, 
photos, selected hot links to other Web sites, and 
occasionally video and sound. That formula is 
now employed on a weekly basis to produce a 
feature package that keys off of events in the 
news. By keeping to a rigorous production 
schedule, the site has expanded considerably in 
the past year, and the dynamic nature of the site is 
a feature that draws a loyal and growing 
following. Another key feature of the site is the 
Cool Science Image gallery, which draws on 
pictures and images from the front lines of 
scientific research. The site also hosts an active 
net forum and the occasional contest, both of 
which boost the site’s drawing power. 

During Years 4 and 5, there have been several 
new developments. The most significant 
development was the move of The Why Files 
production team from the National Institute for 
Science Education to the Graduate School of the 
University of Wisconsin-Madison on July 1, 

1998. The move has placed The Why Files on a 
permanent institutional footing and enabled it to 
expand its activities and collaborations to take 
The Why Files in important new directions. 
Moreover, it has permitted us to expand Why 
Files staff through the addition of a graduate 



student project assistant, a valuable asset that 
has helped the production team maintain its 
rigorous publication schedule and improve the 
site in numerous ways. Inclusion of a project 
assistant also provides important opportunities 
for graduate students who can take the Web- 
content production skills they learn at The 
Why Files to their future employers. 

An important new extension of The Why Files 
now exists in the form of an online 
introductory geoscience course taught through 
the UW-Madison Department of Geology and 
Geophysics. Inspired by geology professor 
and MacArthur Foundation Fellow Jill 
Banfield, and developed in collaboration with 
geology professor Phil Brown, the course was 
taught for the first time in spring 2000 to a 
three-figure enrollment. The department plans . 
to continue offering the course at least once 
every academic year. The course is based 
entirely on Why Files content with added 
material to ensure adequate coverage of basic 
ideas and principles in the geological and 
physical sciences. 

During Years 4 and 5, The Why Files 
continued to maintain a profile as one of the 
most popular and critically acclaimed science- 
related sites on the World Wide Web. By 
providing an in-depth look at the SMET 
content of current affairs and news. The Why 
Files has developed an informational niche 
that attracts a broad audience in terms of both 
demographics and interest. 

Use of the site remains steady. Statistics 
gathered indicate that more than 100,000 
individual computers tap into The Why Files 
in any given month. People from at least 120 
different countries have visited the site. 

The Why Files also continued to enjoy critical 
success and gamers awards and citations at a 
steady pace. Some examples include selection 
as one of the “50 Hottest Sci-Tech Sites” by 
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Popular Science, a site award from SchoolZone 
where teachers serve as judges, a Top Site Award 
from Education Planet, inclusion in Web Feet: 
The Internet Traveler’s Desk Reference, a 
featured site award from Study Web, and a 1999 
Best of the Web Award from HotSheet.com. 
Positive critical reviews of The Why Files site 
appeared in numerous publications, including the 
Chicago Tribune, the Florida Times-Union, the 
Los Angeles Times, Discover magazine, the 
Riverside Press-Enterprise, and 4kids.org, a 
syndicated feature appearing in more than 150 
newspapers in the United States and Canada. 

Anecdotal evidence that The Why Files is used 
and enjoyed by many arrives daily. Some samples 
of readers’ comments: 

• “Serendipity or divine intervention. 
Whatever the reason behind the find, I 
happened upon the Why Files. You’ve 
done a fine job with varied subjects; the 
articles are well done, full of wit, 
educational, and a pure delight (despite the 
sometimes straight truth that may create 
thoughts of becoming a vegetarian). 

• “Way to go Why Files ! Why do I enjoy 
the Why Files so much? Why me. Lord?” 

• “I am doing a science Web site review for 
school and was really impressed with your 
Web site. Especially with the way you 
present both sides of evolution vs. 
creationism. Anyway, you guys are getting 
a great review from me!” 

• “Good site! It doesn't treat me like a 
moron.” 

• “I am a home educator and have just 
‘discovered’ your page. It is great. Thank 
you.” 

The Why Files continues to build collaborations 
with other organizations. The team worked 
closely with the NSF Office of Legislative and 
Public Affairs to produce a package in concert 
with the 1999 and 2000 National Science and 
Technology Weeks. Requests for links are 
routine, and numerous schools, businesses and 



other organizations have established direct 
links to The Why Files over the course of the 
past year. An invited presentation was given 
in Salt Lake City at the first annual Science 
Education on the Internet meeting in the fall 
of 1999. This meeting was open only to a 
select, invited group of science education Web 
site developers from the United States and 
Canada. In addition. Why Files presentations 
and informational sessions were provided at 
Madison’s 1999 Future Fair and the spring 
2000 meeting of the Wisconsin Society of 
Science Teachers. 

With the support of NISE colleagues. The 
Why Files continued to draw together experts 
on the news media and science 
communication as well as the scientific 
community to forge a product that transcends 
the interests of any single institution or 
organization. The work of The Why Files 
continued in that vein as the program made 
the transition from NISE to the University of 
Wisconsin-Madison Graduate School. We 
anticipate that The Why Files will maintain its 
global perspective and ties to NISE and NSF. 

The Research Component 

Over the four-year period that the CMA Team 
has been conducting empirical research, we 
have had a number of accomplishments. 

Below we list our accomplishments in a rough 
chronological order, grouped by major 
research projects. 

Review of Literature on Uses and Cognitive 
Effects of Hypermedia 

The first stages of our research program in 
Year 2 began with an extensive review of the 
literature examining how individuals use the 
technology of hypermedia — the foundation of 
the World Wide Web — to learn. Integrating 
research from fields as diverse as educational 
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psychology and technology, computer science, 
cognitive psychology, geography, library and 
information science, and communication, we 
developed a review of the literature on how 
individuals use hypermedia and its effects on their 
learning. This review, an early version of the 
paper that won a top-three paper award from the 
Communication Technology & Policy division of 
the Association for Education in Journalism & 
Mass Communication (AEJMC), included a 
discussion of theories of information processing 
that could inform research on learning about 
science from Web sites like The Why Files. 
Evidence pointed to a number of important 
variables that might moderate or mediate the 
effects of this new technology, such as the design 
of Web sites and the expertise and goals of users. 
This review of the literature was published by the 
NISE as a Research Monograph. A revised 
version is in press at Communication Yearbook, 
the major outlet in the field of communication for 
extensive literature reviews. 

Audit Trail Analysis of Web Site Navigation; 
Survey of Repeat Users of The Why Files 

In Year 2 we conducted a number of studies to 
evaluate The Why Files Web site. The first of 
these studies unobtrusively tracked natural user 
navigation throughout The Why Files site. During 
three separate two-week periods, we developed a 
data base that allowed us to observe patterns of 
movement within and among science stories on 
the site, as well as movements from outside of the 
site into it, and from inside following links to 
other Web sites. Contrary to the proposed 
advantages of hypermedia systems like the World 
Wide Web, we found that the majority of users 
did not make frequent use of nonlinear navigation 
features within the site or links to additional 
information outside of the site. Instead, they 
tended to follow linear navigation patterns, in 
many ways using The Why Files Web site like a 
traditional print magazine. 



We also conducted a scientific survey of 
repeat users of the site to better understand the 
demographic profile of The Why Files user. 
We found that, much like users of traditional 
print science magazines, and much like the 
users of the Web at the time (1997), repeat 
users of The Why Files tended to be male, 
relatively young, well-educated, and very 
interested in science topics. It was unclear 
from this initial finding whether or not these 
demographic characteristics were due to the 
nature of access to the Web at the time, the 
nature of individuals who tend to be interested 
in science, or some combination of the two. 

The results of these two studies were initially 
published by the NISE as an NISE Brief. A 
much more detailed analysis of these data was 
later published in Public Understanding of 
Science, an international Journal specializing 
in informal science learning research. 

Think-Aloud Analysis of Web Site Navigation 
and Information Processing 

Our finding of linear navigation patterns 
required a more detailed analysis of process 
data to better understand their implications. 
Therefore, in Year 3 we designed and 
conducted a study in which we observed a 
sample of adults using science information on 
the Web and thinking aloud as they did so. 
This study initially allowed us to provide 
advice to The Why Files creative staff about 
how the site was being used and particular 
design features that caused users problems. In 
effect, this study served as a usability analysis 
of The Why Files. Based on a summary of our 
findings, the creative staff of The Why Files 
redesigned the home page of the site as well as 
a number of its other features. 

The results of our think-aloud study also 
enabled us to analyze the information 
processing behaviors of individuals as they 
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navigated through Web sites. We developed a 
coding scheme to categorize thoughts as 
indicating different types of information 
processing behaviors and examined how these 
thoughts varied across a number of variables. The 
results of this research were published in Media 
Psychology, a journal devoted to research on the 
mass media from a psychological perspective. 

Experimental Analysis of Learning from Web vs. 
Print Presentations 

Once we understood the demographics of users of 
science Web sites like The Why Files, and 
understood their navigation behaviors through 
these sites and their information processing 
behaviors during navigation, we expanded our 
research to understand how using the Web to 
communicate science might influence learning. 
Thus, during Year 3 and Year 4 we conducted 
several studies of varying designs using various 
stimulus materials to examine how science 
learning might vary on the Web versus in print, 
holding content constant. These studies also 
manipulated variables such as user motivation 
(i.e., intentional vs. incidental learning) and Web 
site design. Our findings suggest that learning 
may suffer when information is communicated via 
the Web compared to traditional print media. 
However, some Web site designs, particularly 
those that provide both nonlinear navigation 
options and instructional advisement, are not 
significantly worse for learning science 
information than their traditional print 
counterparts. 

Incidentally, during the time of these studies The 
Why Files used just this type of design. 

We are currently submitting the findings of these 
various studies for presentation and publication. 
The first is currently under second review at 
Communication Research, probably the most 
selective journal in the area of mass 
communication. Another has just been submitted 



to Journal of Broadcasting & Electronic 
Media, the major outlet of the Broadcast 
Education Association. 

Experimental Analysis of Elaboration 
Instructions on Learning from Web and Print 

Another series of studies, begun late in Year 3 
and continued through Year 5, examined the 
influence of various manipulations of the 
elaborative processing of individuals as they 
learned about science, primarily from Web 
sites but in some cases also print. Numerous 
studies in education have indicated that 
learning is increased substantially by 
elaborative processing of new information. 
However, these studies tend either to focus on 
very micro-level learning situations (e.g:, 
paired associate learning; short prose 
paragraphs) or to use manipulations that are 
extremely powerful (e.g., semester-long 
learning strategies training). We have 
endeavored to design elaboration 
manipulations that could be incorporated into 
science Web sites to increase elaboration 
among users. Our outcome measures have 
varied from measures of immediate 
recognition in some studies to measures of 
immediate and delayed cued recall in others. 

Our results, which have not yet been fully 
analyzed, suggest that there are a number of 
complexities that may limit the effectiveness 
of elaboration manipulations in real-world 
Web sites. Early findings suggest, for 
instance, that Web sites themselves increase 
elaboration compared to the same content in 
traditional print. This finding is encouraging 
and is consistent with some theories of Web 
effects. However, other findings suggest that 
the elaboration manipulation may not be 
effective on the Web because of a ceiling 
effect. Further, depending on the wording and 
placement of elaboration manipulations in a 
text, learning of the elaborated passages may 
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increase at the expense of other information that 
did not receive the elaboration prompt. During the 
next year we will further analyze these data in 
hopes that we can better understand the role of 
elaboration in learning from the Web. We will 
submit a manuscript or manuscripts to an 
educational technology or educational psychology 
journal. 

Experimental Analysis of Web Site Structure 
Effects on Knowledge Structure 

Our most recent research (Year 5) has examined 
how different Web site designs, specifically the 
use of linear versus nonlinear design structures, 
can influence information processing, content 
knowledge (as tapped by measures of recognition 
and of free and cued recall), and knowledge 
structure (interconnectedness and organization). 

Once these data have been coded and entered, we 
hope to analyze them to determine whether the 
design of Web sites can influence information 
processing, as was suggested in our first think- 
aloud study. In addition, since all our previous 
research using measures of content knowledge 
have found either that print media are superior for 
learning, or at least that Web sites are not superior 
for learning, we have included detailed measures 
of knowledge structure. These measures allow us 
to test the hypothesis that, while Web sites do not 
increase learning of content compared to print, 
different Web site designs can have effects on the 
structure of knowledge, potentially more 
important than effects on the content of 
knowledge. We expect that, without further 
funding, it will take us approximately one year to 
analyze these data and produce conclusions. 

Think-Aloud Analysis of Web Site Structure 
Effects on Navigation and Information Processing 

We have also manipulated Web site structure and 
user goals in a think-aloud study of adults. This 
study allows us not only to assess the impact of 



site design on self-reported information 
processing strategies, but also to determine 
from observational data whether site design 
influences information processing. We will 
also be better able to identify more microlevel 
features of site design that may increase 
beneficial forms (e.g., elaboration) or harmful 
forms (e.g., disorientation) of information 
processing. We will also be able to 
empirically link information processing as 
measured using the think-aloud technique with 
measures of both content knowledge and 
knowledge structure, something our previous 
work was unable to do. We expect that, 
without further funding, it will take us 
approximately one year to analyze these data 
and produce conclusions. 

IIL Staff and Collaborators 

Terry Devitt, Sharon Dunwoody, and William 
P. Eveland, Jr. (Team Leaders), Yael Gen, Sue 
Medaris, HeeSun Park, Jennifer Pearson, 
Darrell Schulte, David Tenenbaum, Amy 
Toburen, Susan Trebach, and Tom Wiggins. 

Product 

The Why Files Web site: 
http://whyfiles.news. wisc.edu/ 

NISE Publications of the Communicating 
with Mass Audiences Team 

Eveland, W. P., Jr., & Dunwoody, S. (1998). 
Surfing the Web for science: Early data on the 
users and uses of The Why Files (NISE Brief 
Vol. 2, No. 2). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 

Eveland, W. P., Jr., & Dunwoody, S. (1999). 
Using hypermedia research to advance the 
study of learning on the World Wide Web 
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(Research Monograph No. 15). Madison: 
University of Wisconsin-Madison, National 
Institute for Science Education. 

NISE-Related Publications of the 
Communicating with Mass Audiences Team 

Eveland, W. P., Jr., & Dunwoody, S. (1997, 
August). Applying research on the uses and 
effects of hypermedia to the study of the World 
Wide Web. Paper presented to the Communication 
Technology & Policy division at the annual 
meeting of the AEJMC, Chicago. 

Eveland, W. P., Jr., & Dunwoody, S. (1997, 
October). Communicating science to the public 
via ''The Why Files” World Wide Web site. Paper 
presented to the International Conference on the 
Public Understanding of Science and Technology, 
Chicago. 

Eveland, W. P., Jr., & Dunwoody, S. (1998). 
Users and navigation patterns of a science World 
Wide Web site for the public. Public 
Understanding of Science, 7, 285-31 1. 

Eveland, W. P., Jr., & Dunwoody, S. (1999, 
August). Examining information processing on 
the World Wide Web using think-aloud protocols. 
Paper presented to the Communication 
Technology & Policy division at the annual 
meeting of the AEJMC, New Orleans, LA. 

Eveland, W. P., Jr., & Dunwoody, S. (2000). 
Examining information processing on the World 
Wide Web using think-aloud protocols. Media 
Psychology, 2, 219-244. 



Eveland, W. P., Jr., & Dunwoody, S. (2000, 
June). A test of competing hypotheses about 
the impact of the World Wide Web versus 
traditional print media on learning. Paper 
presented to the Information Systems division 
at the annual meeting of International 
Communication Association, Acapulco, 
Mexico. 

Eveland, W. P., Jr., & Dunwoody, S. (in 
press). Applying research on the uses and 
cognitive effects of hypermedia to the study of 
the World Wide Web. Communication 
Yearbook #25. 

Eveland, W. P., Jr., & Dunwoody, S. (2000). 

A test of competing hypotheses about the 
impact of the World Wide Web versus 
traditional print media on learning. 
Manuscript submitted for publication. 

Eveland, W. P., Jr., & Dunwoody, S. (2000). 
An investigation of the cognitive mediators 
and moderators of learning from the Web 
versus print. Manuscript submitted for 
publication. 

Millar, T., & Porter, A. (1998, January- 
March). Harnessing the Web for educational 
impact: The science behind the news. 

Envision, 73(1), 20-21. 

Tenenbaum, D. (1997, Summer). Cyber scribe 
tells all: Writing for the web. SEJournal 
[Online]. Available: http://www.sej.org/ 
sejoumal/sej_su97.html 
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Organizational Process Programs 

Cognitive Studies of Interdisciplinary Collaboration 



I. Team Mission 

A major strategy of the NISE has been to use 
interdisciplinary teams to work on the 
Institute’s mission of improving SMET 
education. Within the Institute, experts from a 
variety of disciplines form working teams, of 
various durations, that study significant issues 
and propose and carry out projects related to 
NISE goals. Such interdisciplinary teams are 
ubiquitous in educational research. They are 
increasingly common in industry, government, 
and society in general. The National Science 
Foundation encourages formation of such 
teams through their funding guidelines. For 
example, EHR’s ROLE program and the 
Interagency Education Research Initiative 
(lERI) encourage formation of 
interdisciplinary research teams. 

Although a body of literature and much 
“wisdom of practice” have emerged in recent 
years (e.g., Chubin, Porter, Rossini, & 
Connolly, 1986; O’Donnell, DuRussel, & 
Derry, 1997), we still have little understanding 
of how social and cognitive processes interact 
to drive intellectual growth and construction 
of products in natural interdisciplinary groups 
researching important social problems. Such 
knowledge might lead to better facilitation 
technologies for improving interdisciplinary 
inquiry, which is known to be a difficult 
enterprise. 

The main goal of the Cognitive Studies of 
Interdisciplinary Communication (CSIC) team 
was to better understand interdisciplinary 
collaboration in educational research. To this 
end, our team reviewed research, constructed 
theory, and conducted observational studies of 



NISE’s interdisciplinary teams in action. We 
conducted a two-year video ethnography of 
specified NISE teams and conferences in action, 
collecting data suitable for later analyses by 
researchers, both within and outside the NISE, 
who are interested in developing explanations of 
how groups construct knowledge and how social, 
physical, and cognitive constraints affect their 
work. This report describes our progress on theory 
development, research, and data collection, which 
comprised the bulk of our work during the three- 
year funding period. The CSIC team did not 
operate in years 4 and 5 of the NSF-supported " 
NISE. 

II. Accomplishments 

Development of Foundational Theory 

We grounded our work in a review of scientific 
literature about how individuals and groups carry 
out cognitive activity and how that activity is 
shaped by physical and social contexts. The 
literature review synthesized research from social 
psychology, cognitive psychology, small group 
research, and other fields, highlighting major 
issues influencing cognitive and social processes 
in interdisciplinary teams (O’Donnell, DuRussel, 
& Derry, 1997). Important characteristics of 
effective work groups were described, and origins 
of difficulties within such groups were discussed. 
We determined, however, that much research on 
group work has focused on groups that come 
together only in the laboratory, and that these 
groups differ from true interdisciplinary groups in 
important ways. We concluded that study of 
interdisciplinary inquiry must focus more on 
natural groups that experience special difficulties 
because of institutional constraints, the 
combinations of disciplinary cultures represented. 
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and the implications of disciplinary allegiance 
to problem representations and solution 
strategies. Our review examined several 
examples of interdisciplinary teamwork. 
Methodologies for studying interdisciplinary 
interaction were suggested. To fully 
understand and influence group processes in 
interdisciplinary groups, we argued there is 
need for a broad model of cognition within 
which to interpret the individual-level and 
system-level variables and the interactions 
among these variables. 

Later in the CSIC program such a model was 
proposed, the Knowledge Building 
Community (KBC) model, which is described 
in two publications (Derry & DuRussel, 1999; 
Derry, Gance, Gance, & Schlager, 2000) and 
is just beginning to be adopted by other 
research teams. 

Conferences and Outreach Activities 

To further understanding of interdisciplinary 
teamwork and to help refine research 
objectives for this field of study, the CSIC 
team hosted an invitational interdisciplinary 
conference, cosponsored a national cognitive 
science conference devoted to the theme of 
interdisciplinarity, and participated in a 
community outreach activity featuring 
members of the Sesame Street production 
group. 

Conference on Understanding 
Interdisciplinary Teamwork. Thirty scholars 
attended a small two-day conference in mid- 
November, 1996, entitled “Understanding 
Interdisciplinary Teamwork: Challenges for 
Research and Practice.” Conference 
participants included faculty members of 
NISE, individuals who have special expertise 
or experience pertaining to interdisciplinary 
collaboration and problem solving, and 
representatives of research projects addressing 



the topic of interdisciplinarity. A number of 
questions guided discussion and activities during 
the conference. 

• What is the current theoretical base for 
understanding interdisciplinary collaboration? 

• What is already known about interdisciplinary 
collaboration that can be applied to NISE 
work and beyond? 

• What are the best known methods for studying 
interdisciplinary collaboration? 

• How can current and developing research 
findings contribute to the enhanced 
performance of interdisciplinary teams? 

• What technological enhancements and tools 
might be used to facilitate teamwork within 
the NISE? 

• How might answers to the above questions 
help other research and educational reform 
partnership teams? 

Following an opening address by Dr. Julie 
Thompson-Klein of Kent State University, 
participants began by exploring what is meant by 
the term “interdisciplinarity” and identifying the 
key variables associated with interdisciplinary 
collaboration. The keynote address by Dr. Gavriel 
Salomon of Haifa University, Israel, a noted 
expert in the field, focused the conference on the 
cognitive processes of interdisciplinary teams and 
current understanding of distributed cognition. 
Several case studies, some based on developing 
technologies that support teamwork, were 
presented. Participants then broke up into five 
working groups to explore, in more depth, key 
questions derived from those noted above 
concerning the role of theory, what the state of the 
science is, what the roadblocks are, and the most 
pressing and important questions for researchers 
in the study of interdisciplinary collaboration, as 
well as the potential for creating effective 
environments for collaborative work through the 
use of technology. As the conference closed, 
participants acknowledged their own needs as a 
community of researchers studying 
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interdisciplinary collaboration to meet more 
frequently to share research findings and 
ideas. 

A conference report was prepared based on 
the presentations and working group reports 
and discussions at the conference. The 
conference was very positively evaluated by 
the LEAD Center. 

Cognitive Science Society 1998 Annual 
Meeting. Interdisciplinary scholarship was the 
theme of the Twentieth Annual Meeting of the 
Cognitive Science Society, which took place 
at the University of Wisconsin-Madison, 
August 1-4, 1998. This conference, attended 
by 450 international scholars, was 
cosponsored by NISE and cochaired by 
Sharon Derry, the CSIC team leader. In 
addition to a conference proceedings 
(Gemsbacher & Derry, 1998), the meeting 
produced a set of special thematic papers that 
were organized into a collection and will be 
published by Erlbaum (Derry & Gemsbacher, 
2000 ). 

Sesame Street Unpaved. The CSIC team 
sponsored a graduate symposium with 
members of the Sesame Street production 
group, which visited campus for an 
anniversary performance. The symposium, 
“Understanding Collaborative Work: The 
Children’s Television Workshop as a Case 
Study in Successful Interdisciplinary 
Practice,” took place on September 22, 1998, 
at the University of Wisconsin-Madison, and 
was advertised to faculty and students. 

Data Collection 

We collected observational and interview data 
on three teams within the NISE. These data 
include approximately 1,000 hours of video 
and audio recordings of meetings, 
conferences, and interviews, as well as other 



team artifacts such as email, agendas, and visual 
aids. This database documents the work and 
development of both short-lived and long-term 
interdisciplinary teams within the NISE. About 60 
hours of tape have been transcribed and annotated 
for specific analyses. In the future we will open 
this database to a limited number of outside 
researchers, who will help conduct analyses of 
interdisciplinary teamwork. 

To document our data collection effort, we 
developed a technical manual that describes data 
collection, storage, transcription, and indexing: L. 
A. DuRussel and L. L. Gance, 1997,. Technical 
Manual for the Cognitive Studies of 
Interdisciplinary Collaboration Project. Included 
in the manual is our “master tape list” which 
documents individual pieces of data and status 
information. The manual illustrates the 
organization of the database, what is available in 
it, and how it is being indexed for access. 

Summary of Selected Research Findings 

We conducted studies of two teams in the NISE — 
the Strategies for Evaluating Systemic Reform 
(SESR) team, and a working subcommittee of the 
College Level One (CL-1) team. In both studies 
we examined the genesis and development of 
team knowledge over a lengthy period of time. 

We examined what roadblocks teams encountered 
and what constraints shaped their teamwork. 
Whole working groups were the major units of 
analysis. Our primary method was discourse 
analysis of meeting conversations. These analyses 
focused on collaborative conversations that could 
be observed and recorded, tracing their impact on 
the papers, reports, and other products produced 
by the teams. Our approach was largely 
qualitative in the sense that we descriptively 
analyzed team processes to contribute to theory 
about effective collaborative work and to further 
knowledge about how interdisciplinary 
perspectives influence team functioning. We 
focused less on the private cognitions of 
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individual members, although we did examine 
these selectively through interviews. Detailed 
findings of our work are provided in the 
research reports and publications listed at the 
end of this section. Findings from selected 
studies are summarized briefly here. 

Sociocultural Theory and Interdisciplinary 
Team Building. One study (DuRussel & 

Derry, 1996) examined the validity of 
sociocultural theory (e.g., Wertsch, 1991) as a 
viewpoint for analyzing data from the first few 
meetings of the SESR team. The major goal of 
the analysis was to determine whether the 
Team data “fit” categories and concepts 
proposed by sociocultural theory as lenses for 
examining interaction during the team 
building process. We concluded that 
sociocultural theory did not adequately 
describe major aspects of the team building 
process we observed, leading us to search for 
a more appropriate theoretical viewpoint. 

We found that some knowledge -building 
processes predicted by sociocultural theory, 
namely those of apprenticeship and 
negotiation, did occur occasionally during 
interdisciplinary collaboration. Wertsch’ s 
concept of “voices” was also useful in 
revealing differences between social and 
physical scientists and their strategies for 
finding common ground. However, 
sociocultural theory had only limited 
applicability to the kinds of interdisciplinary 
teams found in the NISE. For example, 
sociocultural theory implies that successful 
interdisciplinary teams operate like 
apprenticeship communities according to a 
process in which high-status old timers induct 
lower-status newcomers. Status differences do 
in fact occur on interdisciplinary research 
teams, but they probably are not associated 
with learning the ropes in a team community. 
Many members of interdisciplinary teams are 
engaged for their expertise and do not have 



novice status as newcomers. Still, in order for 
team members to work together, they must learn 
from one another and develop a common 
language. Although this process is not yet fully 
understood, “apprenticeship” does not appear to 
be an adequate metaphor for describing the 
learning that takes place among high-status team 
members from different disciplines. The need for 
a new theory became evident. Social scientists 
who study group interaction should be interested 
in this study, a critique of a currently popular 
theory as a basis for research on interdisciplinary 
process. 

A Case Study of an Interdisciplinary Team Trying 
to Understand Systemic Reform. This qualitative 
case study by Derry, DuRussel, and O’Donnell 
(1998) employed group information processing 
theory (e.g., Hinsz, Tindale, & Vollroth, 1997; 
Smith, 1994) as a lens for examining the SESR 
team’s dynamics and processes, shedding light on 
how those dynamics and processes likely 
influenced the Team’s effectiveness as a 
knowledge building group. The primary 
contribution of this study is its detailed theory- 
based description of how an interdisciplinary 
team, composed of high-status academic advisors 
constrained by limited time, interacted with a 
team leader and his staff to foster interdisciplinary 
knowledge construction. The analysis highlighted 
difficulties inherent in trying to achieve a truly 
distributed form of information processing under 
this form of team organization. Both limits and 
strengths of the Team’s knowledge construction 
processes were revealed. 

The Team’s difficulties were of three types. First, 
the group had difficulty translating discussions 
into products that reflected those discussions. 
Second, work accomplished at one meeting often 
did not serve as input to subsequent meetings; 
hence the Team had difficulty building on and 
extending accomplishments. Third, both team and 
team leader sometimes rejected recommendations 
and agendas offered by the other. 
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However, the Team leader was a reflective 
manager who was able to adjust team 
organization and functioning as our analyses 
and other feedback became available. 
Moreover, significant knowledge building that 
combined ideas from different disciplines 
contributed by team members did occur, 
primarily within the mind and work of the 
Team leader. The Team leader thus became 
the locus of successful interdisciplinary 
knowledge construction because he valued the 
Team and endeavored to synthesize input 
selectively and thoughtfully. 

Analogical Reasoning and Mental Models in a 
Natural Working Group. A yearlong study, 
conducted by Lori DuRussel as her master’s 
thesis, produced several key findings. First, • 
team members held different mental models of 
the task distribution (who was responsible for 
doing what). Second, misalignments in 
participants’ mental models of the task itself 
contributed to conflicts. Finally, it is crucial 
for interdisciplinary teams to bring team 
members’ mental models of tasks and task 
distribution into some degree of alignment, so 
participants can work together successfully. 

The Team was attempting to analyze students’ 
transcript data to understand how students, 
especially minorities and women, move 
through college curricula in science and 
mathematics fields. The Team used the 
analogy of a “pathway” to characterize their 
viewpoint, and this viewpoint shaped thinking 
about their data analysis. Ambiguities in 
participants’ definitions of what pathways 
were and how they should be investigated 
resulted in conflict. In addition, because 
participants’ models of software’s role in the 
task were not well aligned, the analyst used 
existing software in a way that did not match 
other participants’ expectations, and his work 
was criticized. Even apparent alignments in 
mental models — as in participants’ agreement 



that an inductive analytical approach was 
necessary — did not alleviate the conflict. 

This study identified several factors contributing 
to mental model misalignments. First, as 
expected, individuals’ differing backgrounds 
(including their disciplinary history and 
experience with tools such as software) influenced 
the mental models they brought to the task. In 
addition, the distribution of tasks among team 
members caused each participant’s mental models 
to be shaped by different goals, tools, and task 
experiences. While these misalignments might 
have been resolved through group 
communication, such communication did not 
occur. The lack of articulation of the task 
(including the ambiguous nature of the pathway 
analogy) and the lack of known research 
approaches hampered team members’ attempts to 
work together successfully. Inconsistent 
attendance also may have prevented significant 
alignment from occurring. 

This analysis had several implications for 
management of interdisciplinary teams. Given the 
diverse nature of interdisciplinary teams, some 
degree of mental model misalignment is to be 
expected and possibly desired, since it 
theoretically enhances the wealth of perspectives 
that can be brought to bear on a problem. 
However, it is important that the task be 
negotiated and/or articulated such that participants 
come to share an understanding of the crucial 
steps involved and the expected approach that is 
to be taken. In addition, task distribution should 
be clarified so that all team members are aware of 
who is responsible for what parts of the project. 
Such articulation and clarification requires 
attention to communication. One key to ensuring 
adequate communication is as simple as insisting 
on consistent attendance at team meetings, thus 
ensuring that all team members are influenced by 
the same interactions and increasing the chances 
that participants’ models will evolve in 
compatible ways. 
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Information Resource Coordination 



I. Team Mission 

The mission of the Information Resource 
Coordination (IRC) Team has been to 
examine issues in information technology use 
and knowledge sharing (including information 
technology systems design) to support 
improved information flow for science and 
mathematics education. There have been two 
focus areas of the mission: information flow 
and coordination of NISE projects themselves, 
and the application of research knowledge and 
design engineering to improve information 
sharing in education communities of practice. 

The IRC Team was formed in spring 1999 by 
Associate Director Barrett Caldwell, based on 
his research and teaching interests in 
information and communication technology 
systems engineering. As a result, the IRC Team 
has played a multifaceted role, including 
consulting to other NISE teams (Secondary 
Teacher Education Project, Systemic Reform 
Project), front-office coordination and 
implementation projects, and direct action 
research with educational practitioners. 

The IRC Team accomplishments will be listed 
in three sections: NISE Information 
Coordination Activity, NISE Project Support 
Efforts, and Independent Research and 
Development Tasks. Because of the 
distribution of unique effort by IRC Team 
members, the third accomplishment area will 
be the longest. 

II. Accomplishments 

NISE Information Coordination Activity 

The primary coordination activity in this 
accomplishment area has been focused at 
improving NISE information availability and 



dissemination to the public. By early 1999, the 
online Web presence of the NISE was well 
established, but was difficult for users to recall or 
find through commonly used search engines. In 
addition, the large stock of NISE publications 
were not available for online viewing. The 
activities initiated or supported by the IRC Team 
include the following: 

Registration of simpler Web address. The formal 
Web site URL (universal resource locator) for the 
NISE is http://www.wcer.wisc.edu/nise/. After 
discussions with campus representatives, it was 
determined that there was no conflict of interest to 
register a simpler domain name for the benefit of 
visitors to the site. The name “nise.org” was 
available and was registered as an alternative path 
to the NISE, http://www.nise.org. This address 
takes visitors to the same NISE home page as the 
longer address above. 

Appropriateness of online dissemination. Some 
academic disciplines and professional 
organizations have resisted online publication or 
dissemination of scholarly works due to concerns 
about intellectual property and copyright 
infringement. These concerns had prevented NISE 
from making many of its publications available 
online. A new evaluation of the practice of online 
publication was initiated by IRC Team, and, as a 
result, more NISE Briefs, Reports, and 
Proceedings are available in abstract or complete 
form from the Publications section of the Website. 

NISE Project Support Efforts 

During the 1999-2000 period, IRC Team 
members provided critical information, expertise, 
and research capability resources to two other 
NISE Teams: the Secondary Teacher Education 
Project (STEP: Sharon Derry, Team Leader), and 
the Systemic Reform / Milwaukee Schools Team 
(InfoNet: Norman Webb, Team Leader). Because 
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IRCT members were functionally integrated 
into these other teams, relevant project activity 
will also be summarized in those teams’ 
reports. 

STEP Support. STEP has been focused on the 
development of a “knowledge Web” of 
resources for preservice and inservice teacher 
development in the learning sciences. The 
results of the knowledge Web include a novel 
Web site design to provide case-based 
instructional support for cognitive science and 
learning theory topics in education courses. 
The human-computer interaction (HCI) 
design, implementation, and evaluation of that 
site was conducted in large part by IRC Team 
member Nicole Canty. A conceptual model of 
the overall information resource development 
project is shown in Figure 2. 



STEP required substantial detailed development 
to enable its effective use by educational 
psychology students in their ongoing coursework. 
This development work was begun by both Nicole 
Canty and Jeff Watson and was continued through 
June 2000 by Nicole Canty. 

InfoNet Support. The Systemic Reform/ 
Milwaukee Schools InfoNet project led by 
Norman Webb has identified a critical 
information-sharing problem at the level of 
individual schools’ access, use, and decision- 
making capability of data and resulting 
information produced by the schools themselves. 
This disconnect between the ability to provide 
data annually or semi-annually to the larger 
school district, and the ability to effectively utilize 
local information at shorter time intervals, was a 
primary focus of the support provided by IRC 
Team members Jeff Watson and Susan Zeyher. 



Teacher Professional Development Resources 
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Figure 2. Conceptual model of Teacher Professional Development Information Resources used by 
IRC Team 
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Descriptions of the design of an effective 
InfoNet will be found in more detail in the 
Systemic Reform Team’s reports. However, 
critical elements of an effective InfoNet 
architecture include the availability of data at 
the right time and grain size for local users. 
Systems to provide appropriate data to schools 
or classrooms are not well integrated with 
large scale educational statistics repositories 
for policymaking or funding evaluations. This 
separation of local from general information 
has placed additional burdens on schools to 
modify or invent their own data reporting and 
use systems. The IRC Team has provided 
ongoing assistance to the InfoNet project to 
describe the necessary and desirable 
characteristics of data systems and 
information flow needs for local schools; 
InfoNet reports on these topics are in 
production. 

Independent Research and Development Tasks 

The primary demonstrable outcomes of the 
IRCT are associated with the independent 
project task known as the Science, 
Mathematics, and Engineering Learning 
Technologies (SMELT) project. SMELT has 
involved direct outreach and cooperation with 
teacher practictioners in the Teacher 
Professional Development Institute (TAPPED 
IN), an online “education community of 
practice” hosted by SRI International. In 
addition, SMELT has produced a CD of 
teacher-relevant and “teacher-tested” (i.e., 
submitted by practicing teachers as useful and 
high quality) Web sites with an improved 
search functionality based on an XML 
Metadata architecture (see below). The XML 
Metadata stands as a unique development task; 
however, it is integrated into the CD as a 
method for improving search and selection of 
resources by teachers with slow or limited 
internet access. 



The SMELT project did not simply result in a set 
of tools for practicing teachers. Both the TAPPED 
IN Outreach and SMELT CD Development tasks 
provided opportunities for undergraduate 
engineering students to participate in SMELT as 
part of their educational experience. Members of 
the Society of Women Engineers (SWE) student 
chapter have participated as part of their chapter 
outreach activities; students in HCI design and 
engineering senior design students have 
participated in SMELT design tasks in formal 
classes as part of their design education. 

TAPPED IN Outreach. The IRCT interactions with 
TAPPED IN began as part of STEP project 
interactions in late 1998. Since then, IRC Team 
members have participated in TAPPED IN 
Outreach activities including the TAPPED IN 
Carnival, held annually in July. In spring 1999, 
IRCT members Nicole Canty and Susan Zeyher 
participated in the design of a “virtual suite” for 
TAPPED IN librarian participants from Kentucky. 

At the July, 1999, Carnival, IRCT members hosted 
a “Web site swap” for teachers who were 
interested in sharing or learning about useful Web 
sites for various subjects. Over 75 sites have been 
collected from TAPPED IN participants since then. 
This list has been pared to approximately 45 sites 
primarily focused on science and mathematics 
topics for middle and high school students. These 
sites have served as the prototype content for the 
SMELT CD Development task described below. 

In spring 2000, the UW student chapter of the 
Society of Women Engineers (SWE) began a series 
of outreach activities with TAPPED IN 
participants. These activities include (during the 
academic year) formal “office hours” and 
scheduled question and answer periods for “Ask an 
Engineer” to assist teachers and high school 
students contemplating majoring in science and 
mathematics disciplines. High school to college 
transitions, women in science career role models, 
and general SWE outreach topics are among those 
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that have been discussed (see, for example, 
http://www. tappedin.sri com/cgi- 
bin/calendar/calendar.cgi?file=calendar<Scmon 
th=Aprit). This outreach activity has become 
integrated into SWE ongoing outreach plans 
and is now independent from IRCT 
involvement. 

SMELT CD Development. The SMELT 
development concept of a CD to improve 
teachers’ access and use of online information 
resources began during online discussions 
with teachers in TAPPED IN. Their 
complaints were that there were rarely 
teacher-oriented descriptions of useful 
materials that could be used in the classroom, 
and such materials were difficult to search for 
using commercial search engines. This 
problem is compounded for teachers from 
small and remote schools, who may have 
limited time, bandwidth, or restricted access 
(due to firewalls or other local site 
restrictions) for online searching for Internet 
resources. 

One of the strongest and most frequent 
requests by members of the TAPPED IN 
community has been for a central repository of 
information, including Web sites that are 
suitable for classroom presentations and 
student use. Most teachers do not have the 
time, expertise, or awareness of the resources 
available on the Web to effectively search for 
those resources among the tens of millions of 
Web sites. A repository that can be trusted, 
easily found, and easily used to bring teachers 
quickly to sites they can use is a critical 
technical priority for improving teachers’ use 
of the Web. Such a repository must 
incorporate a human technical support staff 
able to assist teachers in its use, answer 
questions that are raised, and help to maintain 
the repository as a dynamic learning 
environment. 



The SMELT CD, therefore, was designed for 
educators and others interested in quickly and 
effortlessly finding information on the World 
Wide Web about science and mathematics 
education resources. From the users’ manual: 

The CD’s main resource is the “Accurate Search.” 
By clicking on this button, you can search a 
database included in the CD for a number of 
resources available on the World Wide Web. You 
do not need to be connected to the Internet to 
search and evaluate the resources that are 
discussed in the CD database. You should connect 
to the Internet only when you have made your 
choice of resource(s) and want to download them. 
In this way, you can be in any location and look 
for resources without connecting to the Internet 
and dealing with long search times and a 
confusing maze of information.- • 

The SMELT CD design process included major 
development steps in two Industrial Engineering 
design courses. During the fall 1999 semester, two 
project teams in IE 552 (Human Factors 
Engineering Design and Evaluation) were 
responsible for designing the interface, and 
support infrastructure, for a self-sustaining 
SMELT resource. These project teams created the 
initial interface structure for the SMELT CD, 
including installation tools and user guides. 

This first “alpha” prototype was field-tested with 
teachers from Madison, as well as during a site 
visit to Roaring Fork High School, Carbondale, 
CO. Results from this initial site visit were 
brought back to IRC Team members to provide 
practitioner feedback for SMELT improvement. 
Additional feedback was obtained during a spring 
2000 course project in the IE 692 Special Topics 
in Human Factors design course. Usability data on 
existing SMELT interface designs were collected 
from several users, and the results were combined 
with previous usability findings from Roaring 
Fork High School and TAPPED IN designers to 
produce the final SMELT CD prototype. 
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Educational XML Metadata. The major source 
of improvements in information search and 
resource selection in the SMELT CD comes 
from a novel information architecture based 
on the emerging Extensible Markup Language 
(XML) standards. Developing an appropriate 
architecture for use by teachers requires a 
focused development of “metadata” (data that 
describe, classify, and permit search), based 
on the needs and environment of projected 
users. The IRCT focus, unlike most XML 
standards, was on direct use by teacher 
practitioners for in-class resource use, rather 
than on educational administration or research 
applications. In addition, the goal of the IRCT 
activity was to provide teachers with easily 
identified and selected terms for searching 
science and mathematics resources. This direct 
focus on teacher practitioners as end-users, as 
well as hierarchical search architectures for 
science and mathematics, sets the IRCT apart 
from most XML development efforts. 

The IRCT members used as references the 
Dublin Core Education Working Group and 
the national XML standards being developed 
in the U.S. (GEM), Australia (EdNA), and the 
European Union (ARIADNE). By spring 
2000, the IRCT had become (through the 



efforts of Susan Zeyher) a productive and 
contributing member of The Dublin Core 
Education Working Group, providing insights, 
suggestions, and contributions at a level 
commensurate with the national groups. As a 
result, the IRCT XML Metadata can be 
considered a “world-class” architecture with 
capability for increasing development and 
application both at more detailed grains in specific 
science and mathematics disciplines, as well as 
parallel extensions to the social and behavioral 
sciences and humanities. 

Discussions are currently underway to license the 
XML, information architecture, and resource 
coordination functions developed in the SMELT 
project to a Wisconsin company providing 
classrooms with access to live reports and 
continuing availability of mission data from 
NASA missions to the moon, asteroids, and Mars. 

ni. Staff and Collaborators 

Staff: Barrett Caldwell (Team Leader), Armand 
Ardika, Nicole Canty, Virginia “Katie” Emery, 
Christine Lee, Nualwan Narong, Enlie Wang, Jeff 
Watson, and Austen Zuege. 

Fellow: Susan Zeyher 
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Formative Evaluation 



I. Team Mission 

The objectives of the NISE Formative 
Evaluation team evolved over the course of 
the NlSE’s history. The first objective was to 
provide the NISE leadership with real-time 
information about the development of the 
Institute as an organization to guide midcourse 
corrections and thus maximize its goal 
achievement. By the middle of Year Three, 
the NISE team leaders agreed that the NISE 
had evolved to the point where the value of 
intensive formative evaluation reached the 
point of diminishing returns. The formative 
evaluation work was sustained at a low level 
in Years Four and Five. During Year Five the 
formative evaluation objective was completed 
with the production of The Role of Formative 
Evaluation in the Development of an 
Interdisciplinary Academic Center (NISE 
Occasional Paper No. 8). A second objective 
of the Team was to provide formative and 
summative evaluation of the NISE Forums. 
The Team pursued this objective by evaluating 
the First, Second, and Third Annual Forums 
and the Graduate Education Forum. The 
prototype for Forum evaluation having been 
established, the NISE Central Office assumed 
responsibility for this activity as of Year Four. 
During Year Two, the Team began pursuing a 
third objective, which was to draw on its 
social science expertise to produce syntheses 
and proceedings of the Second and Third 
Annual Forums, and the Graduate Education 
Forum. During Year Five, the Team pursued a 
fourth objective — an impact study of the 
College Level One Team’s Field-tested 
Learning Assessment Guide (FLAG). 

NISE's investment in formative evaluation is 
unique among education research centers and 
institutes. After 5 years of what we believe to 
be an especially productive institute, we 



conclude the investment has more than paid for 
itself. A research center of NISE's size and 
challenging scope of work is difficult to create 
and maintain. Time is of utmost importance; 
productivity must begin immediately and increase 
over time. NlSE’s commitment to interdisciplinary 
work across the boundaries of science, 
mathematics, social science, and education added 
to the challenge. The formative evaluation team 
efforts identified key problems as they arose and 
pointed the way to timely solutions. 

II. Accomplishments 

Formative Feedback on the NISE as an 
Organization 

The feedback that the Formative Evaluation Team 
provided during the first three years helped 
Institute leaders understand how various NISE 
constituents were experiencing and viewing the 
NISE teams and the NISE overall (NISE Internal 
Document Nos. 1, 3, 4, 5, and 6). The 
Management Team, Team Leaders, and most team 
members used these reports to make 
organizational improvements. For example, the 
Formative Evaluation Team’s feedback helped the 
NISE leaders 

• redefine their goal for including 
individuals from heterogeneous 
backgrounds, 

• redistribute their budget to fund people for 
a higher percentage of their time, and 

• improve the quality of cross-team 
communication processes by instituting 
retreats and a new members’ orientation 
packet. 

The value of this formative evaluation process is 
described in “The Role of Formative Evaluation 
in the Development of an Interdisciplinary 
Academic Center.” This paper, written during 
Year Five, explores the processes and emerging 
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principles by which one complex, cross- 
disciplinary organization — the NISE — 
developed. In addition, it locates the 
development of the NISE in the context of the 
research literature on other cross-disciplinary 
organizations and makes a case for the utility 
of formative evaluation for the development of 
academic centers. The primary purpose of this 
paper is to serve the NISE, NSF leaders and 
program officers, and others seeking to foster 
productive multidisciplinary academic 
research centers. We are moving with ever 
increasing momentum into a period when key 
problems in science, mathematics, and 
engineering education can no longer be solved 
by people from a single field of expertise, and 
few experts are trained to work in effective 
cross-disciplinary teams. 

Forum Evaluation 

The Formative Evaluation team evaluated the 
first three annual Forums and the Graduate 
Education Forum. These findings appear as 
Report Nos. 2, 7, 8, and 9 of the NISE Internal 
Document Series. These reports enabled the 
organizers of each successive Forum to design 
national events that have improved over time 
and have established a reputation for 
excellence with an increasingly wide national 
audience. 

Forum Planning and Proceedings 

Starting in the second year, the Formative 
Evaluation team began supporting the NISE 
by helping with the planning and analysis of 
the annual Forums. During 1996-97, the Team 
worked closely with the Systemic Reform and 
Interacting with Professional Audiences 
Teams to plan and organize the Second 
Annual Forum. In addition, the Formative 
Evaluation team took major responsibility for 
managing the production of the forum 
proceedings and produced a forum synthesis 



based on participant think pieces (short papers 
prepared by forum participants while on site). The 
synthesis comprises a key component of the 
proceedings and was very positively reviewed. 
During fall 1997, the Team joined the College 
Level One Team’s forum planning group. 

Because the members of the Formative Evaluation 
Team are researchers at the UW-Madison 
Learning through Evaluation, Adaptation and 
Dissemination (LEAD) Center, they brought 
expertise developed through their SMET 
evaluation research work. From February 1998 
through the beginning of Year Four (1998-99), the 
Team produced the Year Three Forum 
Proceedings, working closely with a CL-1 Fellow, 
Elaine Seymour. In addition, throughout Year 
Three, the Formative Evaluation team participated 
on the Graduate Education Forum planning group. 
During Year 4, the Team worked with the leader 
of the Graduate Education Forum to produce the 
Proceedings of this event. 

Formative and Summative Evaluation of the NISE 
College Level One (CL-1) Team's Field-tested 
Learning Assessment Guide (FLAG) Web Site 

The Formative Evaluation team conducted an 
impact study of one of the key products of the 
College Level One Team’s Field-tested Learning 
Assessment Guide (FLAG) Web site. The FLAG 
is an online faculty development resource that 
provides visitors with an introduction to 
assessment, a description of a dozen or so 
alternative classroom assessment techniques 
developed by experts, and various examples of 
these techniques that can be downloaded and used 
(with some revision) in a course. The evaluation 
was designed to determine how well the FLAG 
fits the needs of instructors who teach 
introductory science, math, engineering and 
technology (SMET) courses and to explore ways 
in which instructors report using the assessment 
methods in their courses as well as any new 
understanding of assessment that they develop as 
a result of using the PT.AG. Evaluation data were 
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gathered through individual interviews with a 
representative sample of 50 “reform ready” 
faculty (the audience for which the FLAG is 
designed) who teach biology, chemistry, 
engineering, mathematics, and physics at 
diverse types of institutions of higher 
education. The Team wrote a detailed 
formative feedback report for the College 
Level One Team (see below) and will soon 
submit a manuscript for publication as a paper 
intended for a broader audience. These 
documents address technical aspects of 
participants' use of the FLAG, overall 
reactions to current and future value of the 
FLAG, experience and reactions to the 
individual sections of the FLAG, and 
suggestions for improving the FLAG. The 
LEAD Evaluators also met with the College 
Level One Team web master with a list of 
specific suggestions that came from the 
interview data for improving the FLAG 
design. Many of these changes have been 
made and have been influential in the creation 
of the next College Level One product, the 
Learning Through Technology website. 

III. Staff and Collaborators 

Staff: Susan B. Millar (Team Leader), 

Dianne C. Bowcock, Anne C. Burda, Sarah K. 
A. Pfatteicher, Ramona L. Gunter, Sarah 
Mason, and Susan Daffinrud. 

Products 

NISE Internal Documents 

No. 1 Formative Feedback Report 1.1, 
Baseline Report on the Team Leaders 
Team and Management Team, October 
1995. 

No. 2 Formative Feedback Report 1.2 First 
Annual NISE Forum Evaluation Report, 
April 1996. 



No. 3 Formative Feedback Report 1.3, Baseline 
Report on NISE "Intermediaries," April 1996. 
No. 4 Formative Feedback Report 1.4, Report on 
Non-Team Leader Team Management Team 
Participants, April 1996. 

No. 5 Formative Feedback Report 2.1 Report on 
Perspectives of the Co-Directors, Team 
Leaders Team, and Management Team, 
September 1996. 

No. 6 Formative Feedback Report 2.2, Report on 
Perspectives of the NISE Membership, April 
1997. 

No. 7 Second Annual NISE Forum Evaluation 
Report, March 1997. 

No. 8 Third Annual NISE Forum Evaluation 
Report, April 1998. 

No. 9 Graduate Education Forum Evaluation 
Report, August 7, 1998. 

NISE Publications of the Formative Evaluation 
Team 

Clune, W. H., Millar, S. B., Raizen, S. A., Webb, 
N. L., Bowcock, D. C., Britton, E. D., Gunter, R. 
L., & Mesquita, R. (1997). Research on systemic 
reform: What have we learned? What do we need 
to know? Synthesis of the Second Annual NISE 
Forum, Volume ]: Analysis; Volume 2: 
Proceedings (Workshop Report No. 4). Madison; 
University of Wisconsin— Madison, National 
Institute for Science Education. 

Daffinrud, S. M., & Herrera, O. L. (2000). 
Formative feedback report: Faculty participants' 
experience, reactions to, and use of the Field- 
tested Learning Assessment Guide (FLAG) Web 
site. Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 

Millar, S. B. (Ed.). (1998). Indicators of success 
in postsecondary SMET education: Shapes of the 
future.Synthesis and proceedings of the Third 
Annual NISE Forum (Workshop Report No. 6). 
Madison: University of Wisconsin-Madison, 
National Institute for Science Education. 
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Millar, S. B. (2000). The role of formative 
evaluation in the development of an 
interdisciplinary academic center (Occasional 
Paper No. 8). Madison: University of 
Wisconsin-Madison, National Institute for 
Science Education. 



Millar, T. S., Mason, S. A., Gunter, R. L., & 
Millar, S. B. (1999). Synthesis of the science, 
mathematics, engineering and technology 
Graduate Education Forum (Workshop Report 
No. 7). Madison: University of Wisconsin- 
Madison, National Institute for Science 
Education. 
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Summary 



The Institute’s goals are ambitious. We attack 
head-on three of the most important areas for 
improving SMET education. At the college 
level, we are helping to define an emerging field 
of SMET education R&D; at present, most 
education work in higher education is focused 
on innovation rather than research and is 
scattered. In teacher education, both preservice 
and inservice professional development, we 
seek to reconceptualize the field, identifying 
new and more effective practices and how they 
can be implemented successfully across the 
country. In systemic reform, we seek to set the 
research agenda so that this reform will be 
pursued with increasing effectiveness through 
continuous improvement for decades to come. 
Work in each of these areas enhances work in 
the others, as we focus on their 



interconnections. A strong commitment to 
dissemination insures that our work comes to 
the attention of the NSF, professional 
organizations, practitioners, and other 
researchers. 

At the end of five years, we have launched a 
whole new approach for the continuous 
improvement of SMET education. The goal of 
high levels of SMET literacy for all segments of 
our population is better understood and more 
broadly accepted. New communities of 
scholarship and practice have been established, 
where scientists, education researchers, and 
education practitioners work collaboratively to 
address the enduring problems of SMET 
education, problems that have resisted solutions 
from more narrow approaches. 



Appendix A 

NISE Staff and Collaborators by Team 



Systemic Reform 

Christopher Anderson, Professor of 
Astronomy, UW-Madison 

Steven F. Bauman, Professor of Mathematics, 
UW-Madison 

Vicki M. Bier, Associate Professor of 
Industrial Engineering and Nuclear 
Engineering, UW-Madison 

Jeffery Braden, Professor of Educational 
Psychology, UW-Madison 

Thomas Carpenter, Professor of Curriculum 
and Instruction (Mathematics Education), 
UW-Madison 

Donald Chambers, Researcher, WCER, UW- 
Madison 

William H. Clune, William Voss-Bascom 
Professor Emeritus of Law, UW-Madison 

Janice Downer, Assistant Professor of 
Engineering Mechanics and Astronautics, 
UW-Madison 

Deborah Tepper Haimo, Department of 
Mathematics, University of California, San 
Diego 

Ron Jetty, Project Assistant, Education, UW- 
Madison 

Michael Kirst, Professor of Education and 
Business Administration, Stanford University, 
Stanford, CA 



Michael Knapp, Associate Professor, College of 
Education, University of Washington, Seattle 

Sarah Mason, Associate Researcher, WCER, 
UW-Madison 

Robert Meyer, Senior Scientist, WCER, UW- 
Madison and Lecturer and Research Associate at 
the Harris Graduate School of Public Policy 
Studies, University of Chicago 

Susan Millar, Director, LEAD Center, UW- 
Madison 

Jennifer O’Day, Professor, Department of 
Educational Policy Studies, UW-Madison 

Eric Osthoff, Research Assistant, UW-Madison 

Andrew Porter, Professor of Educational 
Psychology and Director, NISE, UW-Madison 

Senta Raizen, Director, National Center for 
Improving Science Education, Washington, DC 

Alberto J. Rodriguez, Assistant Professor of 
Curriculum and Instruction (Science Education), 
New Mexico State University, Las Cruces 

Judy Roitman, Professor of Mathematics, 
University of Kansas, Lawrence 

Thomas A. Romberg, Sears Roebuck Foundation- 
Bascom Professor in Education (Mathematics 
Education), UW-Madison 

Pat Rossman, Teacher, Elvehjem Elementary 
School, McFarland, WI 

Richard Rossmiller, Professor Emeritus of 
Educational Administration, UW-Madison 
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Bassam Shakhashiri, Professor of Chemistry, 
UW-Madison 

William Tate, Associate Professor of 
Curriculum and Instruction (Mathematics 
Education), UW-Madison 

Christopher A. Thom, Associate Researcher, 
WCER, UW-Madison 

Norman Webb, Senior Research Scientist, 
WCER, UW-Madison 

Paula A. White, Project Manager, NISE, UW- 
Madison 

John Witte, Professor of Political Science, 
UW-Madison 

John Wright, Professor of Chemistry, UW- 
Madison 

Professional Development 

Josefina Arce, Professor of Chemistry, 
University of Puerto Rico, San Juan 

Came Barnett, Regional Director, National 
Council of Supervisors of Mathematics, 
WestEd, Oakland, CA 

Virginia Bastable, Director, Summer Math for 
Teachers, Mount Holyoke College, South 
Hadley, MA 

Cathy Carroll, Regional Director, Math 
Renaissance K-12, WestEd, San Mateo, CA 

Karen Cerwin, Regional Director, California 
Science Implementation Network, Rim Forest, 
CA 

Kathy DiRanna, Mentor Coordinator, 
California Science Alliance, WestEd, Irvine, 
CA 



Mark Driscoll, Project Director, Education 
Development Center, Incorporated, Newton, MA 

Kathy Dunne, Director of Professional 
Development, WestEd, Stoneham, MA 

Joan Ferrini-Mundy, Professor of Mathematics 
Education, University of New Hampshire and 
National Academy of Sciences, Washington, DC 

David Hartney, Managing Director, First Hand 
Learning, Incorporated, Buffalo, NY 

Peter Hewson, Professor of Curriculum and 
Instmction, UW-Madison 

Gregg Humphrey, Project Director, WestEd, 
Willliston, VT 

Susan Loucks-Horsley, Senior Research 
Associate, National Center for Improving Science 
Education, Tucson, AZ (deceased) 

Nancy Love, Dissemination Specialist, National 
Center for Improving Science Education, 
Washington, DC 

Mary Kennedy, Professor, Michigan State 
University, East Lansing 

Barbara Miller, Project Director, Educational 
Development Center, Incorporated, Newton, MA 

Jean Moon, Advisor to the Exxon Education 
Foundation, Exxon Corporation, Groton, MA 

Susan Mundry, National Center for Improving 
Science Education, Andover, MA 

Jan Phlegar, Director, Learning Innovation at 
WestEd, Stoneham, MA 

Lynn Rankin, Director, The Exploratorium, San 
Francisco, CA 
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Ann Rosebery, Principal Scientist, TERC, 
Cambridge, MA 

Susan Jo Russell, Principal Scientist, TERC, 
Cambridge, MA 

Mark St. John, President, Inverness Research 
Associates, Inverness, CA 

Deborah Schifter, Senior Scientist, 

Educational Development Center, Newton, 
MA 

Vernon Sells, Mathematician, Ventures in 
Education, Hespeus, CO 

Jerome Shaw, Senior Research Associate, 
WestEd, Menlo Park, CA 

Rita Starnes, Regional Director, California 
Science Alliance, WestEd, San Mateo, CA 

Mary Stenson, Administrative Assistant, 
Learning Innovations, WestEd, Boston, MA 

Katherine Stiles, Research Associate, National 
Center for Improving Science Education, 
Washington, DC 

Jo Topps, Regional Director, California 
Science Implementation Network, WestEd, 
Long Beach, CA 

Iris Weiss, Evaluator and President, Horizon 
Research Incorporated, Chapel Hill, NC 

College Level One 

Clifford Adelman, Senior Research Analyst, 
U.S. Department of Education/OERI, 
Washington, DC 

Briana Bright, Research Assistant, UW- 
Madison 



Gina Brissenden, CL-1 Team Member, UW- 
Madison 

Aaron Brower, Professor of Social Work, UW- 
Madison 

Ann B. Burgess, Director, Biology Core 
Curriculum, UW-Madison 

James Cooper, Professor of Education, California 
State University, Dominguez Hills 

Patricia Cross, Professor of Higher Education, 
Emeritus, University of California, Berkeley 

Sue Daffinrud, Research Specialist, LEAD 
Center, UW-Madison 

Samuel Donovan, Project Assistant, UW- 
Madison 

Janice Downer, Assistant Professor of 
Engineering Mechanics and Astronautics, UW- 
Madison 

Douglas Duncan, Associate Professor of 
Astronomy and Astrophysics, University of 
Chicago, and National Education Coordinator of 
the American Astronomical Society 

Arthur B. Ellis, Professor of Chemistry, UW- 
Madison 

Roscoe Giles, Professor of Engineering, Boston 
University, Boston, MA 

Jack Husted, Project Assistant, Social Work, 
UW-Madison 

Anthony Jacob, Associate Director of Chemistry 
Learning Center, UW-Madison 

Mary Kennedy, Professor of Teacher Education, 
Michigan State University, East Lansing 
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Louise Liao, Program Director, Coalition for 
Education in the Life Sciences, UW-Madison 

Kate Loftus-Fahl, Science Writer, UW- 
Madison 

Robert Mathieu, Professor of Astronomy, 
UW-Madison 

Catherine Middlecamp, Faculty Associate, 
Chemistry, UW-Madison 

Susan Millar, Director, LEAD Center, UW- 
Madison 

Terrence S. Millar, Professor of Mathematics 
and Associate Dean, Graduate School, UW- 
Madison 

John W. Moore, Professor of Chemistry, UW- 
Madison 

Greg Moses, Associate Dean, Engineering, 
UW-Madison 

Jeanne Narum, Director, Project Kaleidoscope 
and The Independent Colleges Office, 
Washington, DC 

Pamela Robinson, Researcher, School of 
Education, California State University, 
Dominguez Hills 

Gloria Rogers, Vice President, Institutional 
Research and Assessment , Rose-Hulman 
Institute of Technology, Terre Haute, IN 

Walter Secada, Professor of Curriculum and 
Instruction (Mathematics Education), UW- 
Madison 

Karl Smith, Associate Professor of Civil 
Engineering, University of Minnesota, 
Minneapolis 



Mary Beth Stanne, Research Assistant and 
Consultant, University of Minnesota, Minneapolis 

Secondary Teacher Education Project 

Brian Bietzel, Doctoral Student, Educational 
Psychology, UW-Madison 

Barrett S. Caldwell, Associate Professor of 
Industrial Engineering, UW-Madison 

Nicole Canty, Program Assistant, School of 
Engineering, UW-Madison 

Sharon J. Derry, Professor of Educational 
Psychology, UW-Madison 

Kate Hewson, Graduate Student, Educational 
Psychology, UW-Madison 

Jong-Baeg Kim, Doctoral Student, Educational 
Psychology, UW-Madison 

Julia Lee, Graduate Student, Educational 
Psychology, UW-Madison 

Paul Feltovich, Interim Chair, Medical Education, 
Southern Illinois University School of Medicine, 
Springfield, IL 

Cindy Hmelo, Assistant Professor, Program in 
Learning Cognition and Development, 

Educational Psychology, Rutgers University, New 
Brunswick, NJ 

Meg Meyer, Coordinator, Secondary Mathematics 
Teacher Education, UW-Madison 

Susan Millar, Director, LEAD Center, UW- 
Madison 

Roy Pea, Director, Center for Technology in 
Learning, SRI International, Menlo Park, CA 
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Julie Posselt, Project Assistant, Curriculum 
and Instruction, UW-Madison 

Mark Schlager, Director, Teacher Professional 
Development, SRI International, Menlo Park, 
CA 

Rand J. Spiro, Professor of Educational 
Psychology, Michigan State University, East 
Lansing, MI 

Constance Steinkuehler, Project Assistant, 
UW-Madison 

John Street, Project Assistant, UW-Madison 

Christopher A. Thom, Associate Researcher, 
WCER, UW-Madison 

Jeffrey Watson, Project Assistant, UW- 
Madison 

Interacting with Professional Audiences 

Edward Britton, Associate Director, National 
Center for Improving Science Education, 
Washington, DC 

Mary Ann Huntley, Researcher, National 
Center for Improving Science Education, 
Washington, DC 

Susan Mundry, National Center for Improving 
Science Education, Andover, MA 

Senta Raizen, Director, National Center for 
Improving Science Education, Washington, 
DC 

Communicating with Mass Audiences 

Terrance R. Devitt, Senior University 
Relations Specialist, Office of News and 
Public Affairs, UW-Madison 



Sharon L. Dunwoody, Director and Professor of 
Journalism & Mass Communication, UW- 
Madison 

William Eveland, Assistant Professor, School of 
Journalism and Communication, The Ohio State 
University, Columbus 

Yael Gen, Artist, Division of Extension 
Communications, UW-Madison 

Sue Medaris, Artist, University Communications, 
UW-Madison 

HeeSun Park, Graduate Student Research 
Assistant, University of California, Santa Barbara 

Jennifer Pearson, Student Help, University 
Communications, UW-Madison 

Darrell Schulte, Associate Information Processing 
Consultant, UW-Madison 

David J. Tenenbaum, University Relations 
Specialist, Office of News and Public Affairs, 
UW-Madison 

Amy Toburen, Associate Director, Office of 
News and Public Affairs, UW-Madison 

Susan Trebach, Executive Director for Public 
Affairs, University of Illinois, Chicago 

Tom Wiggins, Graduate Student Research 
Assistant, University of California, Santa Barbara 

Cognitive Studies of Interdisciplinary 
Collaboration 

Sharon J. Derry, Professor of Educational 
Psychology, UW-Madison 

Lori DuRussell, Project Assistant, Education, 
UW-Madison 



Laura Lee Gance, Research Specialist, UW- 
Madison 

Jan O’Neill, Educational Consultant, Quality 
Leadership by Design (QLD), Madison, WI 

Mark Schlager, Director, Teacher Professional 
Development, SRI International, Menlo Park, 
CA 

Information Resource Coordination 

Aimand Ardika, Student Help, UW-Madison 

ConneMara Bazley, Teacher, Roaring Fork 
High School, Carbondale, CO 

Barrett S. Caldwell, Associate Professor of 
Industrial Engineering, Purdue, West 
Lafayette, IN 

Nicole Canty, Program Assistant, School of 
Engineering, UW-Madison 

Calvin Chan, Graduate Student, UW-Madison 

William H. Clune, William Voss-Bascom 
Professor Emeritus of Law, UW-Madison 

Sharon J. Derry, Professor of Educational 
Psychology, UW-Madison 

Virginia Emery, Student Hourly, UW- 
Madison 

Christopher Hoadley, Research Scientist, 
Center for Technology in Learning, SRI 
International, Palo Alto, CA 

Christine Lee, Researcher, UW-Madison 

Susan Loucks-Horsley, Senior Research 
Associate, National Center for Improving 
Science Education, Tucson, AZ (deceased) 



Nualwan Narong, Graduate Student, Industrial 
Engineering, UW-Madison 

Christopher Thom, Associate Researcher, WCER, 
UW-Madison 

Enlie Wang, Project Assistant, UW-Madison 

Jefferey Watson, Project Assistant, UW-Madison 

Norman Webb, Senior Research Scientist, 

WCER, UW-Madison 

Paula A. White, Project Manager, NISE, UW- 
Madison 

Austen Zuege, Undergraduate Student, Industrial 
Engineering, UW-Madison 

Formative Evaluation 

Dianne Bowcock, LEAD Center Researcher, 
UW-Madison 

Dianne C. Bowcock, Assistant Scientist, LEAD 
Center, UW-Madison 

Sue Daffinrud, Research Specialist, LEAD 
Center, UW-Madison 

Ramona L. Gunter, Research Training Program 
Fellow, School of Education, UW-Madison 

Olga Lucia Herrera, Research Specialist, UW- 
Madison 

Sarah Mason, Associate Researcher, WCER, 
UW-Madison 

Susan Millar, Director, LEAD Center, UW- 
Madison 

Sarah K. A. Pfatteicher, Assistant Scientist, 
Academic Affairs, UW-Madison 
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Ann Burda Walker, Assistant Professor, 
Teaching and Learning, University of North 
Dakota, Grand Forks 

National Partnership in Advanced 
Computational Infrastructure 
Supercomputer Project^ 

Baine Alexander, Researcher, Engineering, 
UW-Madison 

Julie Foertsch, Associate Researcher, 
Engineering, UW-Madison 

Mike Litzkow, Associate Researcher, 
Computer Science, UW-Madison 

Susan Millar, Director, LEAD Center, UW^ 
Madison 

Greg Moses, Associate Dean, Engineering, 
UW-Madison 

Graduate Science^ Mathematics, 
Engineering, and Technology Education^ 

Clifton Conrad, Professor of Educational 
Administration, UW-Madison 

Christine Golde, Assistant Professor of 
Educational Administration, UW-Madison 

Ramona L. Gunter, Research Training 
Program Fellow, School of Education, UW- 
Madison 

Sarah Mason, Associate Researcher, WCER, 
UW-Madison 



Terrence S. Millar, Professor of Mathematics and 
Associate Dean, Graduate School, UW-Madison 

Leonard Springer, Researcher, UW-Madison 

NISE Leadership and Administration 

Catrina Ahlbach, Administrative Assistant, NISE, 
UW-Madison 

Barrett Caldwell, Associate Professor of Industrial 
Engineering, UW-Madison 

Ji-Eun Kim, Graphic Artist, UW-Madison 

Lynn Lunde, Administrative Assistant, UW- 
Madison 

Robert Mathieu, Professor of Astronomy, UW- 
Madison 

Lois Opalewski, Administrative Assistant, UW- 
Madison 

Andrew C. Porter, Professor of Educational 
Psychology and Director, NISE, UW-Madison 

Ingrid Rosemeyer, Administrative Assistant, 
NISE, UW-Madison 

Deborah Stewart, Senior Editor, WCER, UW- 
Madison 

Paula A. White, Project Manager, NISE, UW- 
Madison 



* The National Partnership in Advanced Computational 
Infrastructure Supercomputer Project was active in 
Year 4 

^ The Graduate Science, Mathematics, Engineering, and 
Technology Education was active in Years 3 and 4 
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Advisory Team 



Vicki M. Bier, Associate Professor of 
Industrial Engineering and Nuclear 
Engineering, UW-Madison 

Michael Corradini, Associate Dean, College 
of Engineering, UW-Madison 

Robert Fitch, Senior Vice President (Ret.), 
Johnson’s Wax, Racine, WI 

LeRoy Lee, Retired Executive Director, 
Wisconsin Academy of Sciences, Arts, & 
Letters, Madison 

Thomas A. Romberg, Sears Roebuck 
Foundation-Bascom Professor in Education 
(Mathematics Education), UW-Madison 



Pat Rossman, Teacher, Elvehjem Elementary 
School, McFarland, WI 

Leona Schauble, Professor of Educational 
Psychology, UW-Madison 

Walter Secada, Professor of Curriculum and 
Instruction, UW-Madison 

Uri Treisman, Professor of Mathematics and 
Director, Charles A. Dana Center for Mathematics 
and Science Education, University of Texas- 
Austin 



NISE Fellows 



Clifford Adelman, Senior Research Analyst, 
U.S. Department of Education/OERI, 
Washington, DC 

Jean-Pierre Bayard, Professor of Electrical 
and Electronic Engineering, California State 
University at Sacramento 

Craig Bowen, Assistant Professor of 
Chemistry and Biochemistry, University of 
Southern Mississippi, Hattiesberg 

Lia V. Brillhart, Coordinator of Engineering, 
Triton College, River Grove, IL 

Cheryl Brown-Kovacic, Mathematics and 
Science Assessment Coordinator, 

Albuquerque Public Schools, Albuquerque, 
NM 

Charles Bruckerhoff, Professor of Science 
Education, University of Connecticut, Chaplin 



Hugh Burkhardt, Professor, Shell Centre for 
Mathematics Education, Nottingham, UK 

Judith Burry-Stock, Professor, Program of 
Educational Research, Area of Professional 
Studies, University of Alabama, Tuscaloosa 

Jeanne Rose Century, Senior Research Associate, 
Center for Science Education, Newton, MA 

Kathy Comfort, Educational Consultant, 
California Department of Education, Sacramento 

Norma Davila, Education Coordinator, Puerto 
Rico Systemic Initiative, San Juan 

Hubert Dyasi, Director, City College of New 
York 

Stephen Ehrmann, Vice President of the 
Teaching, Learning, and Technology Affiliate of 
the American Association for Higher Education, 
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and Director of the Flashlight Program, 
Washington, DC 

Susan Friel, Associate Professor, University of 
North Carolina-Chapel Hill 

Uwe Hilgert, Adjunct Instructor, University of 
Arizona, Tucson 

Andrew Jackman, Associate Dean of Science, 
Mount Hood Community College, Portland, 
OR 

John Jungck, Professor of Biology, Beloit 
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Appendix B 

NISE Conferences, Forums, and Workshops 



Year One 



Conference Title 


Team 


Location 


Date 


College Level One Planning 
Workshop: Articulation, Equity, and 
Literacy Issues 


CL-1 


Madison, WI 


June 23-24, 1995 


Systemic Evaluation Conference 


SESR 


Madison, WI 


January 4-5, 1996 


First Annual Forum 
Professional Development for Science 
and Mathematics Education: Putting 
Knowledge into Action 


PD/IPA 


Arlington, VA 


March 18-19, 1996 


jar Two 








Policy Analysis of Systemic Reform 
Conference 


PASR 


Madison, WI 


July 25-26, 1996 


Understanding Interdisciplinary 
Teamwork Symposium 


CSIC 


Madison, WI 


Nov. 15-16, 1996 


Systemic Reform Seminar I 


PASR 


Madison, WI 


January 15, 1997 


Systemic Reform Seminar II 


PASR 


Madison, WI 


January 17, 1997 


Systemic Reform Seminar III 


PASR 


Madison, WI 


February 4, 1997 


Systemic Reform Seminar IV 


PASR 


Madison, WI 


February 14, 1997 


Second Annual Forum 
Research on Systemic Reform: What 
Have We Learned? What Do 
We Need to Know? 


PASR/ 

SESR/ 

IPA/FE 


Washington, DC 


February 24-25, 1997 


Evaluating Systemic Initiatives 
Conference 


SESR 


Madison, WI 


March 13-14, 1997 


Cooperative Learning in Higher 
Education 


CL-1 


Madison, WI 


May 16-17, 1997 
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ERIC 
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Year Three 



National Education, Outreach, and 
Training Planning/Evaluation Workshop 


NPACI 


Madison, WI 


September 11-13, 1997 


Team Leader Team Retreat 


TLT 


Madison, WI 


September 15, 1997 


Special Emphasis Panel Meeting 


SR 


Arlington, VA 


October 6-7, 1997 


NISE National Advisory Board 
Meeting 


NAB 


Madison, WI 


October 14, 1997 


Professional Development Team 
Conference 


PD 


Washington, DC 


November 14-15, 1997 


Graduate Education Planning 
Conference 


GE 


Madison, WI 


December 8-9, 1997 


Third Annual Forum 
Effective Practices and Their 
Indicators in Reforming Postsecondary 
SMET Courses: Shapes of the Future 


CL-1/ 

IPA/FE 


Washington, DC 


February 23-24, 1998 


Professional Development Miniforum 


PD 


Las Vegas, NV 


April 17, 1998 


Special Emphasis Panel Meeting 


SR 


Arlington, VA 


June 8-9, 1998 


Graduate Education Forum 


GE 


Arlington, VA 


June 29-30, 1998 


Year Four 








NISE National Advisory Board Meeting 


NAB 


Madison, WI 


October 26, 1998 


Fourth Annual Forum 
Evaluation of Systemic Reform in 
Mathematics and Science 


SR 


Arlington, VA 


February 1-2, 1999 


College Level One Institute for 
Technology Planning Retreat 


CL-1 


Madison, WI 


April 1-3, 1999 


Year Five 








Team Leader Team Retreat 


TLT 


Madison, WI 


August 23-24, 1999 
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NISE National Advisory Board NAB Madison, WI 

Meeting 

National Science Teachers Association PD Orlando, FL 
Conference 

Reforming Professional Development 

Fifth Annual Forum SR Detroit, MI 

Diversity and Equity Issues in 
Mathematics and Science Reform: 

What Do We Know? 

What Do We Need to Know? 




October 18, 1999 
April 7, 1999 

May 22-23, 2000 



92 



Appendix C 
NISE Publications 



Research Monographs 
Item 



Code 


Author 


Title 


RMl 


Knapp, M. 


Between systemic reforms and the mathematics and science 
classroom: The dynamics of innovation, implementation, and 
professional learning 


RM2 


Kirst, M., & Bird, R. 


The politics of developing and maintaining mathematics 
and science curriculum standards 


RM3 


Rodriguez, A. 


Counting the runners who don’t have shoes: Trends in 
student achievement in science by socioeconomic status and 
gender within ethnic groups 


RM4 


Tate, W. 


Race, SES, gender, and language proficiency trends in 
mathematics achievement: An update 


RMS 


O’Donnell, A., 
DuRussel, L., & 
Derry, S. 


Cognitive processes in interdisciplinary groups: Problems and 
possibilities 


RM6 


Webb, N. 


Criteria for alignment of expectations and assessments in 
mathematics and science education (in collaboration with the 
Council of Chief State School Officers) 


RM7 


Derry, S., DuRussel, L., 
& O’Donnell, A. 


Individual and distributed cognitions in interdisciplinary 
teamwork: A developing case study and emerging theory 


RM8 


Ridgway, J. 


The modeling of systems and macro-systemic change: 
Lessons for evaluation from epidemiology and ecology 


RM9 


Kahle, J. B. 


Reaching equity in systemic reform: How do we assess 
progress and problems? 


RMIO 


Kennedy, M. 


Defining optimal knowledge for teaching science and 
mathematics 


RMll 


Springer, L., Stanne, 
M. E., & Donovan, S. 


Effects of small-group learning on undergraduates in science, 
mathematics, engineering, and technology: A meta-analysis 
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RM12 


Lee, 0. 


Current conceptions of science achievement in major reform 
documents and implications for equity and assessment 


RM13 


Kennedy, M. 


Form and substance in inservice teacher education 


RM14 


Rodriguez, A. 


Equity through systemic reform: The case of whole-school 
mathematics and science restructuring in Puerto Rico 


RM15 


Eveland, W. P., & 
Dun woody, S. 


A review of educational hypermedia uses and effects with 
application to the study of learning via the World Wide Web 


RM16 


Clone, W. 


Toward a theory of systemic reform: The case of nine NSF 
Statewide Systemic Initiatives 


RM17 


Mundry, S., Spector, 
B., Stiles, K., & 
Loucks-Horsley, S. 


Working toward a continuum of professional learning 
experiences for teachers of science and mathematics 


RM18 


Webb, N. L. 


Alignment of science and mathematics standards and 
assessments in four states 


RM19 


Ghana, C. 


Preservice teacher cohorts and their implications for 
mathematics and science education 


RM20 


Heck, D. J., & 
Webb, N. L. 


Purposes and issues of systemic evaluation in education 


RM21 


Rodriguez, A. J. 


A Cross-case analysis of the Puerto Rico Statewide and the 
Miami-Dade Urban Systemic Initiatives: Promising examples 
of equity in systemic reform 


Occasional 


Papers 




Item 

Code 


Author 


Title 


OPl 


Raizen, S. 


Standards for science education 


OP2 


House, E. R. 


Implementing evaluation findings in government agencies 


OP3 


Clone, W. (Ed.) 


Commentaries on mathematics and science standards 


OP4 


Tobias, S. 


Some recent developments in teacher education in 



mathematics and science. A review and commentary 
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OP5 


Kirst, M., Bird, R., & 
Raizen, S. A. 


Tensions between mathematics and science content 
standards and local politics 


OP6 


Cooper, J. & 
Robinson, P. 


Small-group instruction: An annotated bibliography of 
science, mathematics, engineering and technology resources 
in higher education 


OP7 


White, P. A. (Ed.) 


NISE Fellows program: Feedback from past Fellows 


OPS 


Millar, S. B. 


The role of formative evaluation in the development of an 
interdisciplinary academic center 


Workshop Reports 




Item 

Code 


Author 


Title 


WRl 


Bisgaard, S., et al. 


College Level One: Articulation, equity, and literacy issues. 
The report of a workshop organized by the College Level One 
Team 


WR2 


Webb, N. L„ & 
Heck, D. J. 


Evaluation Strategies Project: A report of the Evaluation 
Questions Conference of the Evaluation Strategies Project 


WR3 


Hewson, P., & 
Loucks-Horsley, S. 


Professional development for science and mathematics 
education: Putting knowledge into action. A synopsis of the 
First Annual Forum 


WR4 


Clune, W. H„ Millar, 
S. B., Raizen, S. A. 
Webb, N. L., Britton, 
E. D., Bowcock, D. C., 
Gunter, R. L., & 
Mesquita, R. 


Research on systemic reform: What have we learned? What 
do we need to know? Synthesis of the Second Annual NISE 
Forum. Volume 1 : Analysis and Volume 2: Proceedings 


WR5 


Gance, L. L. 


Understanding interdisciplinary teamwork: Challenges for 
research and practice 


WR6 


Millar, S. B. (Ed.) 


Indicators of success in postsecondary SMET education: 
Shapes of the future. Synthesis and proceedings of the Third 
Annual NISE Forum 


WR7 


Millar, T. S., et al. 


Synthesis of the science, mathematics, engineering and 



technology Graduate Education Forum 
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WR8 


Webb, N. L. 


Evaluation of systemic reform in mathematics and science: 
Synthesis and proceedings of the Fourth Annual NISE Forum 


Briefs 






Vol. 1, 
No. 1 


Loucks-Horsley, 
S., Stiles, K., & 
Hewson, P. 


Principles of effective professional development for 
mathematics and science education: A synthesis of 
standards 


Vol. 1, 
No. 2 


Webb, N. 


Determining alignment of expectations and assessments 
in mathematics and science education (in collaboration with the 
Council of Chief State School Officers) 


Vol. 1, 
No. 3 


Weiss, I. 


The status of science and mathematics teaching in the 
United States. Comparing teacher views and classroom 
practice to national standards 


Vol. 2, 
No. 1 


Ridgway, J. 


From barrier to lever: Revising roles for assessment in 
mathematics education 


Vol. 2, 
No. 2 


Eveland, W., & 
Dun woody, S. 


Surfing the Web for science: Early data on the users and 
uses of The Why Files 


Vol. 2, 
No. 3 


Kahle, J. B. 


Measuring progress toward equity in science and mathematics 
education 


Vol. 3, 
No. 1 


Mundry, S., & 
Loucks-Horsley, S 


Designing professional development for science and 
mathematics teachers: Decision points and dilemmas 


Vol.3, 
No. 2 


Kennedy, M. M. 


Form and substance in mathematics and science 
professional development 


Vol. 3, 
No. 3 


Meyer, R. H. 


Value-added indicators: A powerful tool for evaluating 
science and mathematics programs and policies 



96 




96 



Internal Documents 

These confidential reports are part of the NISE’s Internal Document series. 



Item 

Code Author Title 



IDl 



ID2 



ID3 



ID4 



IDS 



Millar, S. B., Baseline report on the Team Leaders Team and Management 

Team 

Bowcock, D. C., & 

Burda, A. C. 



Millar, S. B., First Annual NISE Forum 

Bowcock, D. C., & 

Burda, A. C. 

Millar, S. B., Baseline report on NISE “intermediaries” 

Bowcock, D.C., & 

Burda, A. C. 

Millar, S. B., Report on non-Team Leader Team/Management Team 

participants 

Burda, A. C., & 

Bowcock, D. C. 

Bowcock, D. C., Report on perspectives of the co-directors. Team Leaders 

Team, and Pfatteicher, Management Team 
S. K. A., & Millar, S.B. 



ID6 Bowcock, D. C., Report on perspectives of the NISE membership 

Gunter, R. L., 

Millar, S. B„ & 

Pfatteicher, S. K. A. 

ID7 Gunter, R. L., Second Annual NISE Forum evaluation report 

Mesquita, R., 

Bowcock, D. C., 

Pfatteicher, S. K. A., & 

Millar, S. B. 

IDS Mason, S., & Third Annual NISE Forum evaluation report 

Millar, S. B. 

ID9 Gunter, R., & Strengthening graduate education in science and engineering: 

Mason, S. Promising practices and strategies for implementation 
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IDIO 



White, P. A. 



Fourth Annual NISE Forum evaluation report 



ED 11 White, P. A. 



Fifth Annual NISE Forum evaluation report 



NISE Books 

Clune, W. H., Osthoff, E. J., & White, P. A. (2000). Theory and practice of systemic reform of 
mathematics and science education. Manuscript in preparation. 

Derry, S. (2000). Toward a cognitive science of interdisciplinary collaboration. Manuscript in 
preparation. 

Loucks-Horsley, S., Hewson, P., Love, N., & Stiles, K. (1997). Designing professional development 
for teachers of science and mathematics. Thousand Oaks, CA: Corwin Press. 

Lynch, S. (2000). Equity and science education reform. Mahway, NJ: Erlbaum. 

Mintzes, J. J., Wandersee, J. H., & Novak, J. D. (Eds.). (1998). Teaching science for understanding: 
A human constructivist view. San Diego, CA: Academic Press. 

Mintzes, J. J., Wandersee, J. H., & Novak, J. D. (Eds.). (2000). Assessing science understanding: A 
human constructivist view. San Diego, CA: Academic Press. 

Mundry, S., Britton, E., Raizen, S., & Loucks-Horsley, S. (2000). Designing successful professional 
meetings and conferences in education: Planning, implementation, and evaluation. Thousand 
Oaks, CA: Corwin Press. 

Mundry, S., & Loucks-Horsley, S. (2000). Challenges in designing effective professional 
development programs. Manuscript in preparation. 

Mundry, S., Loucks-Horsley, S., et al. (2000). Learning to design: A field book for designing 
professional development for teachers of science and mathematics. Manuscript in 
preparation. 

Stiles, K., Loucks-Horsley, S., et al. (2000). Learning by design. Manuscript in preparation. 

Webb, N. L. (Ed.). (2000). Evaluating systemic reform in mathematics and science. Manuscript in 
preparation. 



Joint Publications 

Adelman, C. (1997). Leading, concurrent, or lagging? The knowledge content of computer science 
in higher education and the labor market (ED Publication No. PLLI 97-8046). Washington, 
DC: U.S. Department of Education. 

Adelman, C. (1998). Women and men of the engineering path: A model for analysis of 

undergraduate careers (ED Publication No. PLLI 98-8055). Washington, DC: U.S. 
Department of Education. 

Britton, E., Raizen, S., Kaser, J., & Porter, A. (2000). Beyond description of the problems: 

Directions for research on diversity and equity issues in K-12 mathematics and science 
education. Washington, DC: National Center for Improving Science Education. 



98 




98 



NISE Web Sites 



Collaborative Learning: http://www.wcer.wisc.edu/nise/cll/CL/default.asp 

College Level One: http://www.wcer.wisc.edu/nise/CLl 

Field-tested Learning Assessment Guide (FLAG): 
http://www.wcer.wisc.edu/nise/cll/flag/default.asp 

Graduate Education Forum: http://nise.wcer.wisc.edu/gradforum/ 

Learning Through Technology: http://www.wcer.wisc.edu/nise/CLl/ilt/default.asp 

NISE home page: http://www.nise.org 

NPACI Supercomputer Project: http://www.npaci.edu/Online 

NSF TechEd99 Workshop: http://www.wcer.wisc.edu/teched99/ 

Student Assessment of their Learning Gains (SALG): http://www.wcer.wisc.edu/salgains/instructor/ 
Secondary Teacher Education Project: http://www.wcer.wisc.edu/step/ 

The Why Files: http://whyfdes.news.wisc.edu 
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Appendix D 

NISE-Reiated Publications 



Adelman, C. (1997). Leading, concurrent, or lagging? The knowledge content of computer science 
in higher education and the labor market (ED Publication No. PLLI 97-8046). Washington, 
DC: U.S. Department of Education. 

Adelman, C. (1998). Women and men of the engineering path: A model for analysis of 

undergraduate careers (ED Publication No. PLLI 98-8055). Washington, DC: U.S. 
Department of Education. 

Boone, W. J., & Kahle, J. B. (1998). Student perceptions of instruction, peer interest, and adult 

support for middle school science: Differences by race and gender. Journal of Women and 
Minorities in Science and Engineering, 4(4), 333-340. 

Bowen, C. W. (2000). A quantitative literature review of cooperative learning effects on high-school 
and college chemistry achievement. Journal of Chemical Education, 77(1), 116-119. 

Clune, W. H. (1998). The national standards in math and science: Developing consensus, unresolved 
issues, and unfinished business. Teachers College Record, 700(1), 6-7. 

Clune, W. H. (1998). The “standards wars” in perspective. Teachers College Record, 700(1), 144- 
149. 

Clune, W. H., Osthoff, E. J., & White, P. A. (2000). Theory and practice of systemic reform of 
mathematics and science education. Manuscript in preparation. 

Clune, W. H., Porter, A. C., & Raizen, S. A. (1999, September 29). Systemic reform: What is it? 
How do we know? Education Week, 79(5), 31 . 

Committee on Undergraduate Science Education, Center for Science, Mathematics, and Engineering, 
and Technology, National Research Council. (1999). Transforming undergraduate education 
in science, mathematics, engineering, and technology. Washington, DC: National Academy 
Press. 

Cooper, J., & Robinson, P. (1998). Small group instruction in science, mathematics, engineering, 
and technology: A discipline status report and a teaching agenda for the future. Journal of 
College Science Teaching, 27(6), 383-388. 

Derry, S. J., DuRussel, L. A., & O’Donnell, A. M. (1998). Individual and distributed cognitions in 
interdisciplinary teamwork: A developing case study and emerging theory. Educational 
Psychology Review, 10, 25-56. 

Derry, S. J., Gance, S. P., Gance, L. L., & Schlager, M. (2000). Toward assessment of knowledge 
building practices in technology-mediated work group interactions. To appear in S. Lajoie 
(Ed.), Computers as cognitive tools II: No more walls. Mahwah, NJ: Erlbaum 

Derry, S. J., & Gemsbacher, M. (Eds.). (2000). The problems and promises of interdisciplinary 
scholarship: Perspectives from cognitive science. (University of Wisconsin-Madison). 
Manuscript in preparation. 

DuRussel, L. A., & Derry, S. J. (1996). Sociocultural approaches to analyzing cognitive 

development in interdisciplinary teams. In Proceedings of the Eighteenth Annual Meeting of 
the Cognitive Science Society (pp. 529-533). Mahwah, NJ: Erlbaum. 

DuRussel, L. A., & Derry, S. J. (1998). Analogical reasoning in a natural working group. 
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Eisenhower National Clearinghouse. (1998). Ideas that work: Mathematics professional 
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hypermedia to the study of the World Wide Web. Paper presented to the Communication 
Technology & Policy division at the annual meeting of the AEJMC, Chicago. 
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''The Why Files'' World Wide Web site. Paper presented to the International Conference on 
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